Document platform evolution

Context

The platform let users upload documents, search their content and organise the knowledge they contained. It had grown around a monolith: many functions shared one application and had accumulated dependencies. Adding capabilities required understanding which other parts might be affected, while users continued to expect improvements.

Document processing also ran at a different pace from the interface. Extracting text from a file or recognising it in an image could take time, fail or need another attempt. The product needed to show that progress and support investigation when a document did not become available.

My role and technical leadership

I took on technical direction for the application. With the team, I decided which capabilities to extract from the monolith, how to structure the new services and which quality improvements should accompany that work. I contributed directly to development and coordinated changes with the people responsible for infrastructure and testing.

I created a common foundation for other developers to build on. It included a service template prepared with infrastructure and a library for repeated data operations. I helped the team adopt them and mentored less experienced developers, explaining the conventions and helping apply them to new features.

My decision-making scope was the application. Infrastructure had its own owners, and priorities were agreed with product. That coordination mattered when a change affected both code and deployment: a component could be finished from a development perspective and still need work to function within the wider system.

Challenges and trade-offs

We chose gradual modernisation, tied to the features the product needed to deliver. Extracting a specific capability let us improve it without waiting to replace the entire application. In return, parts of the monolith coexisted with new services, and their integration needed maintenance throughout the transition.

Separation also added operational cost. Each service needed its own deployment, configuration and diagnosis. Some data dependencies remained shared, so separate code did not provide complete autonomy. We had to weigh the benefit of each extraction against the additional maintenance effort.

The reusable foundation had a similar limit. Sharing repeated operations helped the team work consistently; trying to solve every case through one abstraction could make behaviour harder to understand. I focused the foundation on work common to the services and kept specific rules within each feature.

A shared development foundation

The template established a service’s structure and deployment setup. I built it with infrastructure so new components started from a familiar foundation. Developers could turn their attention to the feature they needed to implement sooner.

The shared library covered repeated operations for creating, querying and updating records. It let developers declare those behaviours consistently, reducing code that would otherwise have been copied between services. I also built a layer for file storage access so each part of the application did not have to repeat the integration.

These components were used within the team. My leadership work included explaining how they worked and supporting developers as they adopted them. Reuse became part of how we developed new capabilities, with fewer accidental differences between implementations.

Asynchronous document processing

One concrete change was extracting document processing into an asynchronous workflow. Receiving a file created work in a queue. Background processes, or workers, picked up that work and extracted text or performed character recognition when the document required it.

Each phase’s status was recorded in the database. The application could distinguish pending, active, completed and failed work without keeping the original request open. This separated receiving a document from the time needed to process it and made progress easier to track.

I also reviewed data access. I found inefficient queries and a single model used for very different needs. I proposed separating structured records from relationships between documents, which required a different kind of query. That proposal was not implemented during my time on the project.

Operation and migration limits

Some processing failures remained difficult to investigate. Jobs could be left incomplete, and execution logs were not always retained. At times, we had to reproduce a problem to obtain enough information. Better status tracking helped diagnosis, but recovery from interrupted executions still needed work.

Retries also did not distinguish well between temporary errors and problems that would recur. Running a job again might recover from a brief interruption or consume resources without changing the outcome. The system needed to handle those situations differently.

Coordination between application and deployment was another difficulty. Configuration changes required separate steps and could fall out of sync with the code. I raised that friction but did not manage to change the working model during my time there. Service extraction progressed, while new product requirements kept arriving, and modernisation remained incomplete.

Results

The team continued delivering features while renewing parts of the platform. Document processing became more reliable, support for formats expanded and it became easier to identify a job’s current state.

The template and shared components reduced repetition and made new services easier to create. I left a foundation that other developers used and supported its adoption, alongside directing the application’s evolution. The improvements were visible in the team’s work, although I do not have measurements that would support expressing them as a percentage.

The platform retained parts of the monolith and operational limitations. Separation of responsibilities had progressed; completing it and improving failure recovery required continued investment.

Separating responsibilities improved the product while it remained in use. Modernisation was still in progress.

What I learned and would change

Today I would require each extraction to include a way to observe the service and recover interrupted work. For asynchronous processing, I would define when a task is acknowledged as complete, what happens if it repeats and how a worker stops without abandoning work in progress. I would also retain the logs needed to investigate failures without reproducing them.

I would make the cost of each new request clearer before adding it to the plan. In this project, I accepted changes without always making explicit what they displaced or how much they complicated the transition.

For disagreements about operation, I would bring smaller proposals with an owner and a concrete way to check the result. I identified friction but did not turn all of it into changes the team could prioritise and carry out.