The Future of OData NxT (Neo)
Few months ago I announced a collaboration with the OData team to start building a new version of OData that can withstand the forever changing requirements in software design and architecture. I got together with Sam Xu and we decided to kick-off a spin-off version of OData.NET that I called OData NxT (Neo)
OData Neo is an attempt to reconceptualize the core capabilities of OData while maintaining the original syntax protocols published by Microsoft back in 2007. The new Implementation will fulfill the following priorities:
This is OData Neo. The next generation of OData Protocol implementation for the .NET ecosystem. This implementation will incorporate three new fundamental design capabilities to OData.NET. These capabilities are:
0. Protocol Agnosticism
The new implementation is aiming towards creating a communication protocol-agnostic library. Which means that OData may no longer be tied up to RESTful APIs communication and can be used broadly across any system-to-system integration.
For instance, if two systems decided to use gRPC or SOAP or any other protocol to communicate, these protocols will not be impediment in establishing that connection while leveraging the powerful capabilities of OData. here’s an illustration of this approach:
The APIs in the above example will simply receive an odata query and turn it into an expression and vice versa. It has absolutely no attachment to any particular protocol of communication which makes it more powerful to integrate with any system.
Protocol-Agnostic OData shall expand the reach of OData to systems that don’t neccearily use REST. But more importantly, this priority will allow OData to become an adapative technology that can withstand the challenge of time in a forever changing industry.
The concept of adaptibility will be driving most of our architecture and design decisions throughtout this project and all other complimantry components that will built to support this effort.
1. Modularity
Modular technology allows the community of engineering to develop and contribute to any platform to make it more useful to fit particular scenarios that may be beyond the priorities of the core engineering team.
Querying data can come in all different shapes and forms. And while the standard OData protocol offers plenty of options to query data in terms of shaping, selecting, ordering and so many other options, it’s still limited in terms of allowing the engineering community to drive larger impact through plug-n-play modules for querying.
OData Neo is aiming to become a marketplace for all things related to data querying over the wire. We want to allow anyone, anywhere to offer more querying capabilities so others can benefit from these capabilities at scale.
At it’s core design, OData Neo shall be completely, and easily be rewritable to allow any engineering team to optimize for performance. We will be avoiding the common mistake of creating un-replaceable, un-rewritable core components that becomes a pain down the road for any team who are trying to push for higher performance, optimizations or even a better implementation.
We have split OData Neo into three main components as follows:
Let’s talk about the components of OData Neo here in detail:
1.0 Substrate
The substrate or the Lakehouse is a new pattern I will be experimenting with to push decoupling of any system components beyond interfaces and dependency injections. Ideally components should be subscribed to one and only one dependency at all times and execute all their communications through that substrate.
The lakehouse pattern will be very similar to queues, except that it allows components to fire-n-observe instead of just fire-n-foreget. This way we allow any component to be plugged into the system even in runtime without impacting the existing flow. Here’s an example:
Assume that our API client is ODataClient which exposes a function Query(string odataQuery). During integration the client would look something like this:
public async ValueTask<IQueryable<T>> QueryAsync(string query) =>
this.dataLikeService.SendObserve(new Message(query), Events.Query);
The substrate will handle communicating with listeners to the Query event and communicate with all the subscribers to that event. This pattern is very similar to the subscriber pattern except that the publisher is waiting on the other side for the listeners to respond.
This way, the API client can send events to any number of subscribers without having to know anything about these subscribers or their contracts. The same thing goes for the subscribers with the publishers. Software engineers now may develop components that plug into the substrate, listen to events in the system and act on it.
1.1 APIs
Exposure components like the exposed APIs from the core library will mainly focusing on two basic functionality:
- String to Expression Capability
- Expression to String Capability
Both are very important to allow engineers to fully control incoming OData queries or generate ones out of simple expressions in the system so they can push it further to other systems.
APIs shall be easily mockable for N-Tier system architecture engineers but also modular to allow for expansion in the future for further capabilities.
1.2 Services
Services are the core business logic of OData Neo. Services will contain all the routines and sub-routines needed to digest and map an OData query into expressions, and strongly typed collections or collection interfaces so they are easily consumable by API developers.
Services can be of three different types, Foundations that hold primitive operations. Processing that holds higher-order business logic and lastly Orchestrations that combine multiple entity types for processing with a unified contract.
1.3 Brokers
Brokers will abstract away native or external dependencies the OData Neo core functions may need so they are replaceable down the road with better dependencies without changing the business logic of the library.
Modularity as a second priority will allow OData Neo Marketplace to expand beyond it’s current capabilities, and we can’t wait to see how our community is going to form their own modules to make OData Neo suitable perfectly for their businesses.
2. Transcendence
Modern software architecture demands certain capabilities in terms of integration. We want to allow OData queries to pass through any number of microservices/lambdas to land on the data-owner service and successfully execute and retrieve a query.
Transcendence in it’s essence is the ability to allow a microservice to receive and pass an OData query to it’s dependencies. This priority is important in terms of adapting to modern design like microservices, serverless and any other patterns that may occur in the future.
But Transcendence also aims for aggregation and control. We want to allow software engineers to develop microservices that can receive one single OData query and execute that query across multiple services or data-owning services. I will call that Query-Fanning and let’s visualize it as follows:
While aggregation is possible today with the current version of OData. It is not as seamlessly as it can be. We aim to make Transcendence a plug-n-play capability. All you got to do as an engineer is to break an incoming expression into smaller expressions with the new OData.Neo library and then post and aggregate responses.
The new implementation of OData will be strictly following The Standard to ensure the software is at the highest quality possible with Test-Driven approach and ongoing pushes for code rubs and pure coding to ensure a systematic intentional engagement of our community in the development of the new technology and it shall also help with the maintainability of the system since it’s being built in the open.
I’m going to document every single step of the way with blogs, videos and pull requests to help newcomers and contributors understand where we were, and where we are today and where we are headed with this technology.
3. Roadmap
Now, let’s discuss the roadmap for OData Neo. This preliminary roadmap will set the initial tentative expectations for deliverables as follows:
There are few things that I need everyone to understand about this roadmap, which are as follows:
- Quality over quantity. If we need more time to make something right, covering more scenarios we will adjust as we go.
- The roadmap is only there to help us and our community of engineers understand what the next step may be. It is a subject to change and enhance as needed.
- This roadmap is very high-level. There will be a further breakdown in our sessions (weekly standup) to breakdown the timeline into more detailed deliverables.
And With the aforementioned priorities we believe that we can explore new realms of software integration where OData can have an even further reach than what it already has today from the smallest APIs and applications to the largest enterprise-size platforms that already serve millions of customers today.
Final Notes & References
Here’s some important information about this new initiative to modernize OData:
- OData Neo does not replace the existing efforts on the current implementation of OData, The OData team will continue to provide more features and new releases of OData while the Neo project effort is going in parallel.
- This is the official repository of OData.Neo – you will find there all documentation, references, blog posts, recordings and most importantly discussions.
- OData Neo is a community effort, we encourage all software design fans out there to join us in shaping the future of OData together. This is a great opportunity to witness the birth of a new technology, step by step and from scratch. My promise is to document every step of the way so you can learn as much as possible from our patterns but also from our mistakes.
- I use initiatives like these to bring the engineering community closer together to build something great. This will happen through pairing sessions like the ones you see here and through discussions as well.
- This effort will be mainly driven by myself and Sam Xu – please reach out if you have any questions or topics of discussion.
- We appreciate the community excitement for this new initiative, and we hope we can deliver something amazing by the end of this year.
Update: After discussion with Sam Xu and Michael Pizzo we decided to call the project OData Neo instead of NxT so it wouldn’t give the impression that it replaces the existing OData effort.
Light
Dark
6 comments
Let me know if/how the Simple.OData.Client nuget team can help!
Thank you, David. I will definitely reach out. Thank you for your support!
I’m a bit concerned/skeptical about the seeming “upfront overengineering” described in this proposal. Being able to use multiple transports makes a lot of sense to me, as does extracting base expression conversion functionality into a simpler, more reusable component, but the whole “substrate”/”custom subscription approach” aspect rubs me the wrong way. Lots of custom patterns without mentioning exactly what these patterns are solving, what scenarios today are problematic that warrant such change, why standard dependency injection-based extension points are not appropriate, etc. This all feels like premature design, which could lead to an overly complex solution that then suffers from maintenance problems.
Also, it is not clear to me why exactly this is a completely separate effort instead of being a gradual migration of existing OData repositories. Has the team decided that the current code is just not extensible/maintainable enough to justify “starting from scratch” like this? OData is a fairly complex beast and there are lots and lots of things already implemented in multiple repositories/packages, yet the team decided to create a brand new, parallel project where everything will be reconsidered and recreated again? Would you mind elaborating a bit more on this aspect and why you think it was a good decision? At first glance, to me it seems like it will just split the community and cause the original project to slowdown even more (if only because Sam will have to split his time even more across all these projects). OData has never been a top priority for Microsoft and this shows with how slow the original solutions have grown (I have first-hand experience on this as I’m in this boat since WCF Data Service was born…). I’m seeing some massive progress towards standardization, support for newer .NET Core features and lots of implementations for missing functionality in the OData spec over the past year or so, and I’d hate to see this momentum halted by a diversion in strategy where it will be years until the new parallel project catches up to current level of functionality.
As for how the subscription/push model is described, this is something I’ve always wanted to see happen. I conceptualized a system once where consumers would send OData queries to a server and “subscribe” to those queries on a callback URL, that would be called whenever there was a new match with the OData query provided. It seems that something like this is an idea the team is pursuing here as well (which I guess would work much better using HTTP2+ or gRPC than my poor-man’s callback URL approach), so for that I’m excited. I’ve heard of the Microsoft internal “Project Reactor” from RX folks and wonder if this would have some overlap with that. I’m not a fan of how “custom” the solution in this article sounds, however. I’d love to hear if the team has considered integrating these solutions with Reactive Extensions / IObservable, instead of coming up with completely new solutions that achieve similar results. RX is not used as much as it should (IMHO) so it would be great to have another framework leveraging its powers and promoting more use out of it.
It is fascinating to see this much support for OData in general though. Even if I’m not thrilled for some of what is being shown here (as it stands, it feels like a bad strategy to me), I do appreciate that OData is getting more love in general. Hopefully you can revert some of the stigma associated with the technology and get more people to dive into it. To this day, OData is considered by many “dead”, “slow”, “poor performance”, “abandoned”, “inflexible”, etc, while everybody keeps citing GraphQL “as the way to go”, even though they are not exactly solving the same problem in all scenarios (and can even be used together). As with my comment regarding RX above, OData is another prime example of an amazing piece of tech that is criminally underrated by the community.
Thank you for your feedback. If I were to distill most of it here it comes down to one question really:
Why the substrate?
This question has two aspects. In reality, why further abstraction. And why a new pattern?
It actually comes down a lot more than that. It’s more like, why innovation?
Every architectural pattern out there goes through a wear and tear process. We try different things, different scenarios and then we decide there’s more that can be done. Simpler, faster, and easier to understand.
Our software industry today compared what it can be will always be in a state of infancy. Doing things, the way they’ve always been done would be a reason why we would be using horses for transportation than cars today. Luckily, this is not the case. It’s not the case because we as engineers have the urge to continue to look further into a more advanced, brighter, and innovative future. This is the case here with OData Neo.
It’s an attempt to create an R&D branch for an existing technology and explore the possibility of reconceptualizing OData as a protocol as a whole. It’s also risk-free because it doesn’t impact nor utilize the existing engineers on the main project. But also, it leverages the community to build what best suites all of us together. Which has more social impact than the technology that will eventually be a legacy one way or another.
It goes even further, by documenting the entire experience of picking up an existing technology, no matter what it is and rethinking it completely. Baking into the process concepts like trying a completely brand-new patterns never used, innovating new ways, and hopefully bringing more results.
OData Neo has multiple purposes, it’s a POC for strategic management to evolve existing technologies. But it’s also a technology that is built rewritable by design. As engineers we tend to build things to last forever. I’m experimenting with building things are super easy to understand that it also becomes easier to rewrite.
And this is where the other aspect of the Lakehouse comes in. Allowing truly plug-n-play pattern to encourage engineers everywhere to develop their own components with the least effort of understanding how the whole system works and turn OData into a marketplace for ideas that is not limited by a communication protocol, a framework, language or technology.
In terms of OData popularity and usability, the numbers of downloads on our nuget packages seem to show some sort of stability. And that stability is not a good thing. Because our engineering community continues to grow, therefore our numbers should also grow. This doesn’t seem to be strongly the case today. A rebranding, reconceptualizing and re-architecture attempt might reveal some of the reasons why we are not seeing the growth we expect.
Every pattern we use today was at some point in time “custom”. Then it proved itself to be useful and successful so it became just a pattern. This is usually how innovation works. The Lakehouse pattern is brand new, it combines multiple existing concepts that I’m hoping it would produce a modular truly agnostic solution. Is it going to work? We are going to have to walk the walk to find out.
We will rebuild a community-powered version of OData and rethink everything. It may be a bad strategy and may be not so bad. We will never know until we take our shot and see what comes from the other side. The one thing I know for certain, is that one way or another we are all going to learn something. And I can’t wait to see what that “something” is.
I hope this sheds some lights on your concerns, and thank you for the feedback.
Thank you very much for the detailed response Hassan.
In regards to the innovation aspect of the proposal, I actually don’t think that’s bad per-se at all. I agree with you, in that if nobody tries something new, then we’d all be stuck with old tech and old patterns forever. However, at the same time, I think most patterns “emerge” out of a problem, and are not created upfront. The description above and your reply indicate to me that we are coming up with this new pattern before any code was written, before any specific problem was tackled. This is what I meant by upfront: we seem to be doing huge upfront design even before truly understanding the scope we want to tackle. That’s the part that worries me, as this tends to backfire once requirements change along the way and you have the now inflexible architecture, created for a different purpose, that needs to still be leveraged in a cumbersome way. Rather, it would be best to start with the basics and then adapt as you go. Don’t take me wrong: I love OData and I’d love for you to be successful, but I’m still not convinced about the upfront design strategy being employed here.
Regarding my concerns over development resource allocation being split between OData and OData Neo, I’ll take your word for it, but I see that Sam is already involved in this new project in some capacity. As you well know, the OData team is extremely small right now: a few months ago, it was basically Sam working by himself while a couple other folks were still ramping up. If what you say is true, and the original OData team won’t take a cut because of this new branch, then I don’t see the problem. Lots of interesting innovation came from R&D like CodeContracts, so I’m totally in favor of these initiatives.
There is an aspect of my comment that you didn’t address, however, and that is in regards to the “start from scratch” strategy vs incrementally moving existing OData libraries towards the goal. Could you elaborate a little bit about that? It is not clear to me why the team decided to create a brand new branch instead of just extending the existing libraries with satellite projects, for example. This will surely result in the new team having to replicate a huge amount of features that have already been implemented in the original libraries, and nobody likes having to fix the same bugs and go through the same trouble all over again to get to the same end result. The OData spec is fairly complex, and even the existing libraries are severely lacking in features, always playing a catch-up game to ensure all the new functionality is represented. We’ll be creating a new parallel OData branch that will have to reimplement all the same spec definitions. Wouldn’t it be better to focus instead on the existing library, and ensuring it matches the protocol vision first?
And finally, I’d also like to hear a bit about my comments regarding RX and Project Reactor. I’m fine with code being custom, but only when an existing robust solution is not already in place. You mentioned asynchronous notifications and subscriptions in the main post, which seems to fit very nicely with the proposal of these existing solutions. Are you considering leveraging them, or are you really pursuing a custom solution on that front as well? As I mentioned, this could be an interesting way to boost the visibility of RX, which from what I can tell suffers in a similar way to OData in terms of audience and usability.
In any case I hope this effort is fruitful, as I certainly don’t want to see yet another great Microsoft piece of tech die. There seems to be a considerable focus on “community” and “social aspects” there, which I personally don’t find to be essential for a new tool (i.e. the “odata marketplace” dream seems quite farfetched to me), but it could certainly serve as great PR for advertising OData.
Thanks Juliano.
It’s true patterns do emerge as problem arises. We have learned from the past from current implementations at the enterprise level including OData that three priorities/problems needs to be achieved/solved. These are the protocol-agnostic state, modularity and Transcendence.
These issues do exist today with some workaround solutions. And we are trying to streamline these issues and turn them into opportunities that can inspire and influence newer patterns in software design and architecture.
You don’t actually have to create patterns after writing code, you can create ones as a result of reading the existing code and how current patterns influence our development.
Being able to develop a module with the least amount of knowledge needed to know how to plug it into an existing (running) system is one of these problems we are trying to solve at a higher level. OData Neo is an opportunity to implement the substrate/Lakehouse to see if this can be a viable solution for other projects to adapt and implement.
Solving that problem with a fire-n-observe solution seems very promising, but also very complex problem to solve. Doing things the old way isn’t going to help us achieve that level of modularity and further the dependency injection portion beyond traditional contracts.
So, in summary, the pattern of Lakehouse does emerge from existing issues that I saw in the modern design and architecture in general and we are just going to give it a shot. And see how far it goes. The good news is that like I said it doesn’t impact current production flow of OData, this is our R&D flow – where we go and take risks and try all the new ideas we could come up with.
I will try to address the rest of your comment soon, my apologies because of the time restriction. And I appreciate your feedback.