Getting Hands-on: Autonomous Coding

by

Getting Hands-on with Autonomous Coding

Cognition’s Devin has been making waves in the software engineering community. With Cognition’s robust backing, including notable investors like Peter Thiel, and a hefty valuation of $2 billion as of July 2024, Devin is positioned as a groundbreaking tool. While eagerly waiting for my turn on the Devin waitlist, I decided to give OpenDevin, its open-source pendant, a shot. My goal was to see how it fared with a complex, real-world task: pulling information from the German commercial register. The search on the site is the interface they needed to provide based on a legislation to open access to German public data and reflects the state of digitisation of the German public sector quite well.

The Task: Extracting Data from the German Commercial Register

The task I assigned to OpenDevin was straightforward yet complex enough to test its capabilities. Here were the steps:

  1. Access the German commercial register website.
  2. Enter a company name in the search field.
  3. Select an option from a radio button group.
  4. Click the search button.
  5. Wait for the navigation to complete.
  6. Click the download link for the document.

Execution and Challenges

With my expectations tempered, I set a rate limit of $10 per day for API access and initiated the task. While I monitored the process with half an eye during a phone call, I noticed a flurry of terminal outputs and repeated task attempts. By the time I returned, OpenDevin had burned through $7 of GPT-4 API costs within 15 minutes and ultimately encountered an error.

Upon closer inspection, the primary issue was with the setup of Selenium and ChromeDriver. OpenDevin tried various approaches and made some progress but couldn’t fully resolve the problem. In contrast, when I manually set up the initial configuration, it took only a few minutes to fix, allowing me to proceed seamlessly.

Reflection and Insights

This experience with OpenDevin was both grounding and inspiring. Despite the hurdles, the potential for such AI-driven tools is evident. Running OpenDevin in debug mode might provide better insights into the setup issues and help overcome initial barriers quickly. Considering the vast amount of code and well-specified requirements in repositories worldwide, tools like OpenDevin and the real Devin have the potential to become indispensable assets for software developers.

The Future of AI-Driven Software Engineering Tools

Looking ahead, the evolution of AI in software engineering is promising. Devin and OpenDevin, with continued improvements and human interaction to guide them, could revolutionize how we approach complex coding tasks. These tools will likely become more proficient and efficient, making them valuable companions for developers.

In conclusion, while OpenDevin’s first run had its setbacks, the experiment underscored the importance of perseverance and fine-tuning. As these AI agents evolve, their integration into the software development process will undoubtedly enhance productivity and innovation, heralding a new era in software engineering.