“Throttled” async download in F#

by

I think there must be a better way to find out that a file is not available than using a timeout. I’m not exactly sure, but is there some way to make it throw an exception if a file cannot be found? Then you could just wrap your async code inside try .. with and you should avoid most of the problems.

Anyway, if you want to write your own “concurrency manager” that runs certain number of requests in parallel and queues remaining pending requests, then the easiest option in F# is to use agents (the MailboxProcessor type). The following object encapsulates the behavior:

type ThrottlingAgentMessage = 
  | Completed
  | Work of Async<unit>

/// Represents an agent that runs operations in concurrently. When the number
/// of concurrent operations exceeds 'limit', they are queued and processed later
type ThrottlingAgent(limit) = 
  let agent = MailboxProcessor.Start(fun agent -> 
    /// Represents a state when the agent is blocked
    let rec waiting () = 
      // Use 'Scan' to wait for completion of some work
      agent.Scan(function
        | Completed -> Some(working (limit - 1))
        | _ -> None)
    /// Represents a state when the agent is working
    and working count = async { 
      while true do
        // Receive any message 
        let! msg = agent.Receive()
        match msg with 
        | Completed -> 
            // Decrement the counter of work items
            return! working (count - 1)
        | Work work ->
            // Start the work item & continue in blocked/working state
            async { try do! work 
                    finally agent.Post(Completed) }
            |> Async.Start
            if count < limit then return! working (count + 1)
            else return! waiting () }
    working 0)      

  /// Queue the specified asynchronous workflow for processing
  member x.DoWork(work) = agent.Post(Work work)

More Related Contents:

  1. Asynchronous vs synchronous execution, what is the main difference? [closed]
  2. How do I run Asynchronous callbacks in Playground
  3. Implement C# Generic Timeout
  4. How to make a simple JSONP asynchronous request in Angular 2?
  5. What is the purpose of async/await in Rust?
  6. asynchronous and non-blocking calls? also between blocking and synchronous
  7. How do I conditionally return different types of futures?
  8. is node.js’ console.log asynchronous?
  9. How to add a timeout to a function in Python
  10. What is the concrete type of a future returned from `async fn`?
  11. What is the difference between launch/join and async/await in Kotlin coroutines
  12. How does await async work in C# [closed]
  13. How to make an asynchronous Dart call synchronous?
  14. Async/await and parallel in C# [closed]
  15. Testing with React’s Jest and Enzyme when simulated clicks call a function that calls a promise
  16. How to implement a Future or Stream that polls an async fn?
  17. Asynchronous vs synchronous execution. What is the difference?
  18. Angular 2 – Displaying async Object data from promise
  19. Calling an async method from a constructor in Dart
  20. Calling setState in render is not avoidable
  21. How can I store an async function in a struct and call it from a struct instance?
  22. What is the difference between Asynchronous calls and Callbacks
  23. Are JavaScript ES6 Classes of any use with asynchronous code bases?
  24. How can I use the async keywords in a project targeting.net 4.0
  25. MySQL server has gone away – in exactly 60 seconds
  26. NSURLConnection sendAsynchronousRequest can’t get variable out of closure
  27. React – Display loading screen while DOM is rendering?
  28. Make WPF Image load async
  29. A Simple AJAX with JSP example
  30. Timeout function using threading in python does not work

Leave a Comment