Big Data Read

Some articles to handle big data architectures:

http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36632.pdf

http://incubator.apache.org/drill/drill_overview.html

http://www.drdobbs.com/database/applying-the-big-data-lambda-architectur/240162604

http://mdavey.wordpress.com/2013/12/17/lambda-architectures-in-finance/

http://data-informed.com/lambda-architecture-can-analyze-big-data-batches-near-real-time/

http://jameskinley.tumblr.com/post/37398560534/the-lambda-architecture-principles-for-architecting
Big Data Reactive Way

Lamda Acrhitecture in General

 

Building high throughput, low latency systems

So you want to build an HA low latency system – here is how you get started

    Understand that you have to change your thinking about what you know already.
    Sort your threading issues out a disruptor pattern architecture is the way to go, read the technical paper.
    Know your Java concurrency esp. phasers.
    Build for failover – take a tip from the let it crash, Event Sourcing, and concurrent systems thinking
    Look at whats is available in the world of concurrent development frameworks.
    Concurrency comparisons, disruptor, akka, threads, etc
    Know what is on the tech. radar so you can plan accordingly for change and make your product last longer than 12 months.
    Lastly, employ the right talent (see sidebar), you only get one chance to shine! If you do you can reap the rewards.

Still need help get in touch.

← Back

Thank you for your response. ✨

Building asynchronous applications in 2013 – same shirt different pattern

So most of the finance community shunt data from one place to another – and they need to shunt this data in realtime and asynchronously. In the old days you had to write your own handlers, concurrency controls, thread handlers and non-blocking queues. In a nutshell they all follow the reactor pattern.

Well not any more – you have now have the open source toolset available at your finger tips – providing the boilerplate and letting you concentrate on doing the actual business work. The following is what is available in your toolkit, making sure you avoid callback-hell.

    Akka
    Vert-X
    Spring Reactor

So take your pick and run with it – they each have their strengths for the job at hand.

Collateral Management and EMIR, DFA Regulation

This is where the focus will be for the next 12 months.

Collateral Management is the task of reducing credit risk. A good paper to indicate the challenge faced by institutions is provided by Foxeye Consultants report, “Benefit from one of the biggest opportunities in 2013: Collateral Management”.

A summary by Sungard – “Are Your Collateral Management and Hypothecation Methods Ready for Prime Time?”

Regulation for EMIR is now required – be ready for 2014, the EMIR timetable is out.

Finally, differences between DFA and EMIR Regulatory Requirements.

Metamaterials and Cloaking – Cool Stuff

Realizing the world’s first 3D cloak of invisibility, engineering professor Andrea Alú explains how the discovery of metamaterials is pushing technology beyond conventional limits, producing vastly new opportunities beyond what nature can offer.
This emerging technology offers extensive applications in bioscience, energy, defense and plenty more we can only imagine from here.

12 months of work to reach this goal – DTCC FX Regulatory – 28th Feb 2012

So 12 months of work and it comes down to this date – to keep a bank in business. Reporting FX trades in realtime to maintan Dodd Frank Act.


2013-02-27 23:59:03,322 363566054 [clearingControlEventExecutor-1] INFO org.perf4j.TimingLogger log - start[1362009542481] time[841] tag[PERF:ClearingControlEventPollingAdapter.doWork(clearing_dtcc_equity)] message[Sent MQ controlevents DTF_DISP_READY successcount=2/2 Count=2 Duration=841(ms) Rate=2.378/sec DurationOne=0.42 sec]

2013-02-28 00:00:36,575 363659307 [clearingControlEventExecutor-1] INFO org.perf4j.TimingLogger log - start[1362009633292] time[3283] tag[PERF:ClearingControlEventPollingAdapter.doWork(clearing_dtcc_fx)] message[Sent MQ controlevents DTF_DISP_READY successcount=122/122 Count=122 Duration=3283(ms) Rate=37.161/sec DurationOne=0.027 sec]

It is one of many major milestones that the FX market place requires as many instituitions realise technology is the key to deliver business agility and success.