Building a RESTful Web API in Clojure: a new approach

Malcolm Sparks <mal@juxt.pro>

CTO, JUXT

REST & Me

restlet (Java)

2009

2012

WebMachine

plugboard

bidi

2015

Liberator

2021

rest.guide

pick

2018

reap

jinx

apex

  • Client/Server
  • Stateless
  • Cache
  • Uniform interface
  • Layered System
  • Code-On-Demand

REST Recap (Constraints)

Uniform Interface

By applying the software engineering principle of generality to the component interface, the overall system architecture is simplified and the visibility of interactions is improved. Implementations are decoupled from the services they provide, which encourages independent evolvability. The trade-off, though, is that a uniform interface degrades efficiency, since information is transferred in a standardized form rather than one which is specific to an application's needs.

Fielding, Roy Thomas. Architectural Styles and the Design of Network-based Software Architectures. Doctoral dissertation, University of California, Irvine, 2000.

  • Broad reach
    • Communicate to many different types of clients (both human and machine)
  • Discoverability/Introspection
    • what they can do (options)
    • what they did wrong (error codes)
  • Performance
    • Exploit caches (private and shared)
  • Security
  • Options

REST Benefits

From https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation

Resources & Representations

  • Semantically correct
  • Not overly restrictive
  • Possible to adapt to various situations
  • Performance

Goals

  • Lots of extra coding?
  • There's an awful lot to understand and get right
  • That's the point of this talk - to provide you with a step-by-step guide to implementing REST

Costs

Semantic Correctness

HTTP Flowchart

From https://github.com/basho/webmachine/blob/master/docs/http-headers-status-v3.png

RFC 7231: Semantics and Content

RFC 7232: Conditional Requests

RFC 7233: Range Requests

RFC 7234: Caching

RFC 7235: Authentication

RFCs

Inversion of Control?

The 'Hollywood' Principle

Don't call us, we'll call you

Avoid frameworks!

No callbacks please, hang on to your threads!

The 15 Steps

  • Content negotiation
  • Methods
  • Correct status codes
  • Request/response headers
  • Conditional requests
  • Security headers
  • and more!

15 Steps

But 15 Steps?

  • Ring middleware
  • Synchronous (single thread)
  • Can be implemented with interceptors
    • e.g. Pedestal, Sieppari
  • Can we made to work asynchronously

Code samples

(defn ring-middleware
  [opts]
  [wrap-ring-1-adapter
   wrap-healthcheck
   #(wrap-initialize-request % opts)
   wrap-service-unavailable?
   wrap-log-request
   wrap-store-request
   wrap-error-handling
   wrap-method-not-implemented?
   wrap-locate-resource
   wrap-redirect
   wrap-find-current-representations
   wrap-negotiate-representation
   wrap-authenticate
   wrap-authorize
   wrap-method-not-allowed?
   wrap-initialize-response
   wrap-security-headers
   wrap-invoke-method])

Step 1

Initialize the request state

(defn wrap-initialize-request
  "Initialize request."
  [h]
  (fn [req]
    (let [extended-req
          (into
           req
           {:start-date (java.util.Date.)
            :request-id (java.util.UUID/randomUUID)
            :uri
            (str "https://"
                 (get-in req [:ring.request/headers "host"])
                 (:ring.request/path req))})]

      (h extended-req))))

Step 2

Service Available?

Step 3

Method Implemented?

(defn wrap-method-not-implemented? [h]
  (fn [{:ring.request/keys [method] :as req}]
    (when-not (contains?
               #{:get :head :post :put :delete :options
                 :patch
                 :mkcol :propfind} method)
      (throw
       (ex-info
        "Method not implemented"
        (into
         req
         {:ring.response/status 501
          :ring.response/body "Not Implemented\r\n"}))))
    (h req)))

Step 4

Locate the resource

  • Return a Clojure map
    • Resource 'state'
    • Resource 'configuration'
  • Use multiple strategies, e.g.
    • Routing
    • Pattern matching
    • Database lookup
    • Dynamic generation

Step 4: Locate the resource

(defn locate-resource [req]
  (case (:ring.request/path req)
    "/weather"
    {:id :weather
     :description "Today's weather"

     ;; Resource state
     :weather/precipitation 35
     :weather/outlook "Overcast"
     :weather/temperature 16

     ;; Resource configuration
     :http/methods #{:get :head :options}}))

(defn wrap-locate-resource [h]
  (fn [req]
    (h
     (assoc
      req
      :resource (locate-resource req)))))

Step 5

Redirect

(defn wrap-redirect [h]
  (fn [{:keys [resource]
        :ring.request/keys [method] :as req}]
    (when-let [location (:http/redirect resource)]
      (throw
       (ex-info
        "Redirect"
        (-> req
            (assoc :ring.response/status
                   (case method (:get :head) 302 307))
            (update :ring.response/headers
                    assoc "location" location)))))
    (h req)))

Step 6

Current Representations

  • Only for GET, HEAD and PUT
  • Return a collection of representations
  • For a representation, use a Clojure map

Step 6: Current Representations

  • A representation includes
    • payload
    • representation metadata
      • content-type
      • content-length
  • If none found, throw a 404 (not for PUT)

Step 6: Current Representations

(defn current-representations [req]
  (let [res (:resource req)]
    (case (:id res)
      :weather
      [{:http/content-type "text/html;charset=utf-8"
        :http/content-language "en"
        :http/content-length 210}

       {:http/content-type "text/html;charset=utf-8"
        :http/content-language "es"
        :http/content-length 228}

       {:http/content-type "application/json"
        :http/content-encoding "gzip"
        :http/content-length 189}])))



(defn wrap-find-current-representations [h]
  (fn [{:ring.request/keys [method] :as req}]
    (if (#{:get :head :put} method)
      (let [cur-reps (seq (current-representations req))]
        (when (and (#{:get :head} method) (empty? cur-reps))
          (throw
           (ex-info
            "Not Found"
            (into req
                  {:ring.response/status 404
                   :ring.response/body "Not Found\r\n"}))))
        (h (assoc req :cur-reps cur-reps)))
      (h req))))

Step 7

Content Negotiation

(require '[juxt.pick.alpha.ring :refer [pick]])

(defn negotiate-representation
  [{:ring.request/keys [method] :as req} cur-reps]

  (let [{rep :juxt.pick.alpha/representation
         vary :juxt.pick.alpha/vary}
        (pick req cur-reps {:juxt.pick.alpha/vary? true})]

    (when (#{:get :head} method)
      (when-not rep
        (throw
         (ex-info
          "Not Acceptable"
          (into
           req
           {:ring.response/status 406
            :ring.response/body "Not Acceptable\r\n"})))))

    (cond-> rep
      (not-empty vary) (assoc :http/vary vary))))
(defn wrap-negotiate-representation [h]
  (fn [req]
    (let [cur-reps (:cur-reps req)]
      (h (cond-> req
           (seq cur-reps)
           (assoc
            :selected-rep
            (negotiate-representation req cur-reps)))))))

Step 8

Authenticate

  • Check request to authenticate the 'subject'
  • The subject might include:
    • Username
    • Roles
    • Authentication method
  • If necessary, redirect to an 'authorization server' (identity provider)

Step 8: Authenticate

(defn authenticate [req]
  ;; Check Authorization header
  ;; Check request for session cookies
  ;; Extract subject from JWT, or from session store
  ;; Redirect to an identity provider if necessary
  )

(defn wrap-authenticate [h]
  (fn [{:ring.request/keys [method] :as req}]
    (if-let [subject (when-not (= method :options)
                       (authenticate req))]
      (h (assoc req :subject subject))
      (h req))))

Step 9

Authorize

  • Approve or Deny the request
    • 401 = authentication required
    • 403 = authenticated but still no
  • Lots of options here:

Step 9: Authorize

;; Allow read access to all resources tagged as PUBLIC
[[request :ring.request/method #{:get :head :options}]
 [resource :classification "PUBLIC"]]

;; Allow read access to all resources tagged as 
;; INTERNAL for logged in users
[[request :ring.request/method #{:get :head :options}]
 [resource :classification "INTERNAL"]
 [subject :crux.db/id _]]

Step 10

Method Allowed?

(defn join-keywords
  [methods upper-case?]
  (->> methods seq distinct
   (map (comp (if upper-case? str/upper-case identity) name))
   (str/join ", ")))

(defn wrap-method-not-allowed? [h]
  (fn [{:keys [resource]
        :ring.request/keys [method]
        :as req}]
    (let [allowed-methods (set (:http/methods resource))]
      (when-not (contains? allowed-methods method)
        (throw
         (ex-info
          "Method not allowed"
          (into
           req
           {:ring.response/status 405
            :ring.response/headers
            {"allow" (join-keywords allowed-methods true)}
            :ring.response/body
            "Method Not Allowed\r\n"}))))
      (h (assoc req :allowed-methods allowed-methods)))))

Step 11

Perform the Method

(defn wrap-invoke-method [h]
  (fn [{:ring.request/keys [method] :as req}]
    (h (case method
         (:get :head) (GET req)
         :post (POST req)
         :put (PUT req)
         :patch (PATCH req)
         :delete (DELETE req)
         :options (OPTIONS req)
         :propfind (PROPFIND req)
         :mkcol (MKCOL req)))))
(defn GET [{:keys [selected-rep] :as req}]

  (evaluate-preconditions! req)

  (let [{:keys [body]} selected-rep]
    (cond-> (assoc req :ring.response/status 200)
      body (assoc :ring.response/body body))))
(defn POST [{:keys [resource] :as req}]
  (let [rep (receive-representation req)
        req (assoc req :received-representation rep)
        post-fn (:post-fn resource)]

    (if (fn? post-fn)
      (post-fn req)
      (throw
       (ex-info
        "No post-fn function!"
        (into
         req
         {:ring.response/status 500
          :ring.response/body "Internal Error\r\n"}))))))

(defn PUT [{::site/keys [resource] :as req}]
  (let [rep (receive-representation req)
        req (assoc req :received-representation rep)
        put-fn (::site/put-fn resource)]

    (if (fn? put-fn)
      (put-fn req)
      (throw
       (ex-info
        "No put-fn function!"
        (into
         req
         {:ring.response/status 500
          :ring.response/body "Internal Error\r\n"}))))))

(defn DELETE [{:keys [crux-node uri] :as req}]
  (crux.api/submit-tx crux-node [[:crux.tx/delete uri]])
  (into
   req
   {:ring.response/status 202
    :ring.response/body "Accepted\r\n"}))
(defn OPTIONS [{:keys [resource allowed-methods] :as req}]
  (-> (into req {:ring.response/status 200})
      (update :ring.response/headers
              merge
              (:options resource)
              {"allow" (join-keywords allowed-methods true)})
      ;; Also, add CORS headers for pre-flight requests
      ))
  • RFC 7232: Conditional Requests
  • Exploited by caches to re-validate
  • Mitigates 'lost-update' problem
  • Uses representation metadata
    • last-modified
    • entity tags
    • ranges

Step 11: Evaluate pre-conditions

(defn evaluate-preconditions
  [{:ring.request/keys [headers method] :as req}]
  (when (not (#{:connect :options :trace} method))
    (if (get headers "if-match")
      (evaluate-if-match req)
      (when (get headers "if-unmodified-since")
        (evaluate-if-unmodified-since req)))
    (if (get headers "if-none-match")
      (evaluate-if-none-match req)
      (when (#{:get :head} (:ring.request/method req))
        (when (get headers "if-modified-since")
          (evaluate-if-modified-since req))))))
  • No content length? 411
  • Bad content length? 400
  • Content length too large? 413
  • No body? 400
  • Unacceptable content-type? 415
  • Unacceptable content-encoding? 409
  • Unacceptable content-language? 409
  • Content-Range header? 400
  • Got here? Slurp and return the body

Step 11: Receive request payload

Step 12

Prepare the Response

  • Create the response, adding headers to reflect the 'representation metadata'

Step 12: Prepare the Response

(defn respond
  [{:keys [selected-rep body]
    :ring.request/keys [method]
    :as req}]

  (cond-> req
    true (update
          :ring.response/headers
          assoc "date" (format-http-date (java.util.Date.)))

    selected-rep
    (update :ring.response/headers
            representation-headers
            selected-rep body)

    (= method :head)
    (dissoc :ring.response/body)))

(defn wrap-initialize-response [h]
  (fn [req]
    (respond (h req))))
(defn representation-headers [headers rep body]
  (into
   headers
   {"content-type" (some-> rep :http/content-type)
    "content-encoding"
    (some-> rep :http/content-encoding)
    "content-language"
    (some-> rep :http/content-language)
    "content-location"
    (some-> rep :http/content-location str)
    "last-modified"
    (some-> rep :http/last-modified format-http-date)
    "etag" (some-> rep :http/etag)
    "vary" (some-> rep :http/vary)
    "content-length"
    (or (some-> rep :http/content-length str)
        (when (counted? body) (some-> body count str)))
    "content-range" (:http/content-range rep)
    "trailer" (:http/trailer rep)
    "transfer-encoding" (:http/transfer-encoding rep)}))

Step 13

Security Headers

Step 14

Error Handling

(defn wrap-error-handling [h]
  (fn [req]
    (try
      (h req)
      (catch clojure.lang.ExceptionInfo e
        (let [{:ring.response/keys [status] :as exdata}
              (ex-data e)]
          (log/errorf
           e "%s: %s"
           (.getMessage e) (pr-str exdata))
          (respond
           (merge
            {:ring.response/status 500
             :ring.response/body "Internal Error\r\n"
             :error (.getMessage e)
             :error-stack-trace (.getStackTrace e)}
            exdata
            {:selected-rep (error-representation e)})))))))

Step 15

Logging

(defn log-request! [{:ring.request/keys [method] :as req}]
  (assert method)
  (log/infof
   "%-7s %s %s %d"
   (str/upper-case (name method))
   (:ring.request/path req)
   (:ring.request/protocol req)
   (:ring.response/status req)))

(defn wrap-log-request [h]
  (fn [req]
    (doto (h req) (log-request!))))

Further Reading

  • https://www.rest.guide
  • https://github.com/juxt/site
  •  RFC 7230-7235

Further Reading

https://www.rest.guide

Discussion, Q&A