Web crawling reliability

Context

A product used an automated browser to visit web pages and analyse their behaviour. Jobs ran against external sites whose loading times and structures could change without notice. Some jobs stayed active without making progress; others failed without leaving enough information to explain the outcome to the user.

The system was already in production and had little test coverage. We needed to resolve incidents while requests continued to arrive, with limited room to reorganise the architecture. I started by understanding how each job began, how it reported progress and where information about a failure could be lost.

My role and technical leadership

I took technical ownership following a handover from the previous developer. I became the point of contact for investigating incidents and making maintenance decisions. I coordinated changes to expected behaviour with product and worked with customer support to understand the problem each user was seeing.

That responsibility included tracing a failure from symptom to cause, choosing the scope of a fix and implementing it. I introduced improvements to execution control and tests for problematic behaviours. I also proposed architectural changes to address limitations that local fixes could not resolve.

The broader proposals needed shared priority and resources. They did not become an execution plan during my time there, so I concentrated on improving the existing product’s reliability and continuing feature delivery.

Challenges and trade-offs

A crawler needs to decide when a page is ready to analyse. Some content appears after the initial load, and some components are embedded in separate documents. Waiting too long extends execution; moving on too early can miss relevant information. I adjusted the navigation strategy and expanded support for those cases, while retaining the system’s heuristic approach.

I introduced time limits for operations that could wait indefinitely. Choosing them meant accepting that some slow pages could end in an error, while preventing jobs from consuming resources without progressing. I strengthened error propagation so the product could report a meaningful outcome.

A complete rewrite would have delayed fixes that users already needed. Incremental changes let us address those problems within a smaller scope, although dependencies and restrictions from the inherited architecture remained.

Tracking each execution

The system received a request, passed it to processing and ran the analysis in the background. The process controlling the browser reported progress and results to the application. Operations could fail at several points before completing that path.

I reviewed input validation and error handling between those stages. A local exception needed to be reflected in the overall job status. Otherwise, part of the system could have stopped while the product continued to show the execution as active.

During investigations, I inspected stored state and looked for the last known step in execution logs. This helped distinguish a failure at startup from one during navigation or result reporting. It narrowed the investigation before I changed code.

A job that never finished

One incident involved an active job that had stopped progressing. I followed its state and logs until I found an invalid input address. The error occurred while preparing execution, before analysis began, and was not correctly reflected in the status the user saw.

I added early validation when creating the request and a defensive check before execution. I also strengthened error handling so a failure at that stage would not leave a job apparently active. I reviewed which errors could be retried and which should end the job with an identified failure.

A retry made sense for a temporary interruption. Invalid input needed correction before another execution. Distinguishing those cases helped avoid pointless repetition and retain an outcome that could be investigated.

Testing and architectural evolution

I gradually introduced unit and end-to-end tests. Unit tests checked specific behaviours; end-to-end tests ran crawls and inspected their results. Cases that had caused incidents helped us verify corrections and reduce the risk of introducing the same failure again.

Tests against real pages provided variety, though those pages could change between runs. Adjusting navigation and improving support for dynamic content increased consistency in the cases investigated. The system still depended on the behaviour of external sites.

I also identified that progress reporting and result handling could put too much load on the application. I proposed separating that traffic more clearly from interactive use and reducing the cost of handling large results. That evolution remained pending; the improvements delivered focused on execution control, error handling and tests.

Results

Stalled jobs and silent failures became less frequent. Status information gave us more context for investigations and helped us provide clearer responses to incidents. Executions became more consistent in the behaviours we had corrected.

Reliability improved within the product’s normal development work. My contribution combined technical ownership of an inherited system, coordination with product and customer support, and concrete changes to its behaviour. There was not enough historical measurement to quantify the reduction in failures, so the results are qualitative.

When something fails, there is context to investigate and act.

What I learned and would change

I would define reliability measures earlier: what proportion of jobs reaches a final state, how long they take and which stage fails. This would have helped connect fixes to their effect on product use and agree priorities with better information.

I would add a set of controlled pages to check repeatable behaviours alongside tests against real sites. That would make it easier to distinguish our own regressions from changes in the pages being analysed.

I would divide architectural proposals into small deliveries with a cost, an owner and a verifiable improvement. I resolved incidents and strengthened the system, but some structural limitations remained without a plan to address them.