How the Hungarian Government automated reporting processes
Developing an efficient government administration by automating processes and reporting for key decision makers.
"Government executives can access reports on the web, in real-time, providing them with vital, timely information for better decision making. The interactive, web-based reports enable executives at various levels to drill down into the data as needed."George AndorCEO and Founder, iCode (KNIME Partner)
The capital or county government offices are provincial state administrative branches of the Hungarian government and are the largest administrative units at a provincial level. There are twenty offices, which are there to not only improve and coordinate the implementation of government policies, but also adjust decisions from central government to better meet the needs of their regions. These offices are led by a government commissioner and coordinated and supervised by the Prime Minister’s office.
Developing an efficient and customer-friendly public administration
Operating in 174 cities and in the 23 districts of the capital city, these offices have more than 1,000 competencies and responsibilities. One of them is to offer and promote a service-provider state to citizens and businesses. This initiative aims to develop an efficient and customer-friendly public administration, resulting in an improved quality of public services. Factors contributing to this include reorganizing central public administration, reforming district offices, modernizing and reducing bureaucracy, and creating so-called “Government Windows.” These windows provide information and services to citizens in more than 1,500 different types of administrative cases. To support the reform initiatives, the government mandated the development of the Government Offices Integrated Management System, part of which is the Management Information System.
The capital and county government offices, along with the district offices and Government Windows, form the largest organization in Hungary. Their diverse functions are supported by many enterprise-sized IT systems including operations management, human resources management, financial planning and accounting, customer service and customer calling, as well as document and records management. There are also many specialized administrative systems, for example social insurance.
There were a few shortcomings of the existing solution. There was no unified reporting dashboard that was accessible to all stakeholders. There was also no automated information flow, meaning reports took weeks to be created and delivered. Lastly, the data sources were not interconnected. The Prime Minister’s office often requests information from administrative staff in the form of consolidated and aggregated reports. Previously, this was a manual, labor-intensive, and slow process.
A no-code solution to produce automated, web-based reports
A solution that could automate the collection, aggregation, and consolidation of the data from the separate systems, and produce both web-based and printed reports with current information on demand was needed. Furthermore, the government’s own data analyst team needed to be able to develop new reports when needed, without coding.
A simple, low-cost data warehouse was built on several PostgreSQL databases. Because KNIME workflows can handle multiple databases as one, this enables data from diverse sources to be segmented across the databases, resulting in easy scaling on inexpensive hardware. KNIME Server and a set of special KNIME workflows are used as an automated ETL tool for extracting the data from the different systems daily, and then transforming, aggregating, and storing them in the data warehouse. The SQL, SOAP, and REST interfaces of KNIME are used to connect to the various systems, while the scheduled execution is used for evenly spreading and timing the data transfers to off-peak hours.
A KNIME Server installation consists of one KNIME Server and four distributed KNIME Executors, which are connected through a RabbitMQ. The entire system is duplicated in a geo-redundant, high availability setup, and the deployment and configuration are automated with Ansible. This ensures that the development, test, and production systems are identical, and allows new environments to be installed within minutes if needed.
The Management Information System is integrated into an existing government Network Access Manager (NAM) system, providing OAuth authentication, a directory, an SSO system, and a reverse proxy, which protects all system resources behind it. It’s also integrated into a central logging system, which collects all KNIME Server, database, and OS system logs. A customized KNIME WebPortal meets government guidelines and is completely in Hungarian. A custom system-monitoring software checks the system health at the service level, which is where the KNIME REST API is extremely useful.
The bulk of the project was the development of 45 KNIME workflows, which produced 80 interactive WebPortal reports and their PDF versions using the BIRT integration. These reports are two versions of the same workflow, which was a requirement of the project so that different users could interact with the data. Recreating the reports in BIRT required a significant amount of work but improved the overall project deliverables and user satisfaction. The email sending capability of KNIME Server automatically distributes the PDF reports.
An interesting and unique feature of the Management Information System is the in-workflow authorization mechanism. This restricts the scope of the information available for the user during the workflow execution and user interaction, based on privileges stored in the NAM directory server, which is queried using KNIME REST nodes.
Results
With this solution, all data reporting processes have been converted from manual to automatic. Government executives can access reports on the web, in real-time, providing them with vital, timely information for better decision making. The interactive, web-based reports enable executives at various levels to drill down into the data as needed. For example, from the country level down to counties, cities, and lower levels of the organization. This was previously not possible without support from analysts and has therefore empowered executives to be more independent with the data.
Why KNIME?
KNIME Software was chosen because it offers out-of-the-box solutions to almost all requirements - and was easily extensible where it didn’t. Specifically, KNIME:
- Enables non-coder data analysts to create new reports on demand
- Is an excellent and flexible report distribution platform that also offers user authentication and authorization
- Acts as an automated ETL tool
- Enables solutions to be scaled and reproduced when needed, and all steps are self-documenting
- Integrates with a multitude of external tools, which is invaluable in a project where many different data sources must be connected
- Is extensible and integrates seamlessly with other tools and technologies, which ensures that unforeseen problems can be easily resolved
Because of the free, open source, and lightweight nature of KNIME Analytics Platform, it was easy to evaluate and verify it against the project requirements. The software learning curve was low, and it was possible to develop a proof of concept quickly and efficiently. The KNIME team provided expert support with the few issues related to the new and more complex features such as OAuth authentication and SSO integration, which added another level of confidence to the final solution. The transparent pricing of KNIME Server is fair – particularly the unlimited number of consumers with access to KNIME WebPortal with Server Large. This was essential for giving executives access to the reports. Lastly, the partner program meant that iCode, a KNIME Partner, could provide the required expertise and support due to their in-depth knowledge of data solutions in the government context.