DataCleaner

We've released  MetaModel version 1.7.3 today! This release is a minor bugfix release which continues to stabilize and build a firm foundation for this data access framework. The improvements added in version 1.7.3 are: Handling of errors in ... [More] Excel formulas. If an invalid calculation is made in a formula, MetaModel will simply return the formula as a string. Handling of invalid Excel formulas. Similarly if a formula in excel is simply invalid (eg. using symbols that do not exist), MetaModel will simply return the formula as a string. Improved handling of fetch size caluclation for JDBC queries. In particular some queries can be identified as only returning a single row (eg. COUNT(*) queries and the like). Such queries will get an appropriate fetch size of 1. All  Javadoc API Documentation warnings where fixed. We hope you will enjoy the new version of  MetaModel! [Less]

Another release of  DataCleaner sees the light of day today! Although this is not a major release, but a minor one, it does ship some quite nice stabilizing improvements and minor enhancements to the UI. Enhancements in 2.1.1: Added a ... [More] search/filtering text field on the datastores list. This enables you to quickly find your datastore if you have registered more datastores than available on the screen. Reference data for country codes was added to the standard distribution, thanks goes to  Graham Rhind for providing these. Added a horizontal scroll bar to the data previewing windows of there are more than 10 columns. Ability to add an extension package with new functionality in the Options dialog at runtime. More focus on extensions will follow in the upcoming releases. We've exposed an early preview of our Command-Line Interface (CLI) by allowing you to invoke the application with the "-usage" parameter which will show the CLI options. Added number formatting options to the "Convert to Number" transformer. Bugfixes in 2.1.1: Fixed an out-of-memory issue when querying tables with a LOT of columns (150+). Fixed an issue that cause the "Limit analysis" check box to not be checked correctly when a job was re-opened after saving. Not really a bugfix as it was never an official feature, but now we support restoring user preferences (the userpreferences.dat file) from previous versions of DataCleaner. Thanks to everyone involved in the making of this release of DataCleaner. DataCleaner 2.1.1 is available as a traditional download or as a Java Web Start application on  the downloads page. Keep in touch with your  feedback to the application on  the forums. [Less]

Due to the attention that we've received from our release earlier this week, we've quickly collected a few improvements to MetaModel 1.7 that where small but valuable. This is why we are today releasing MetaModel 1.7.1! The release contains these ... [More] new items: Various minor improvements in the  API Documentation. Fixed a minor bug that occurred when CSV headers are configured to be read from an unexisting line number. Added the capability to calculate an appropriate FETCH_SIZE for JDBC/database queries. This feature allows for better memory management when used with databases that take advantage of eager buffering. Please turn to  the MetaModel website for downloads! [Less]

Today is the day that  MetaModel version 1.7 has been released! The focus of the new version was to bring additional configurability and handling of special corner cases into the framework. In addition we've also improved performance and fixed a few ... [More] minor issues. For a full list of new stuff, take a look at the  What's new in MetaModel 1.7? page. As always you can  download MetaModel as a distributable or get it from the Maven repositories: <dependency> <groupId>org.eobjects.metamodel</groupId> <artifactId>MetaModel-full</artifactId> <version>1.7</version> </dependency> We look forward to getting your reactions and feedback. Please let us know if you use MetaModel and we will also be happy to add you to the  list of projects that use it. [Less]

We're happy to announce the release of  DataCleaner 2.1! This is a quite significant release and something that we hope users will recognize as a step forward from the 2.0 versions. The major news in DataCleaner 2.1 are: There was a lot of ... [More] work done on the user interface (see  media page): We decided to remove the left-hand side window containing environment configuration options. Instead all these options have now been moved to the job building window so the user only has to focus on a single window for all the interactions needed to build a job. The welcome/login dialog has also been removed in favor of a more discrete panel that can be pulled in or hidden from the main window. Datastore selection and management is considered the first activity in the application, which is why it is also the first step to handle in the main window. You can now stop jobs in case you decide to change something before it is done. Bar and line charts were added to a lot of the analysis result screens, including String analyzer, Number analyzer, Date/time analyzer and Weekday distribution (see  media page). All "preview data" windows now contain paging controls so you can move backwards and forwards in the data set. Most common database drivers (MySQL, PostgreSQL, Oracle, MS SQL Server and Sybase) have been added to a default set of drivers. Configuration of the Quick analysis function in the Options dialog. Various minor bugfixes. Transformer for extracting date parts (year, month, day etc.) from date columns. We hope you enjoy DataCleaner 2.1. Please head over to the  downloads page to get it! [Less]

We're happy to announce today the release of MetaModel 1.6. The new version of  MetaModel has three focus points: A new datastore type has been added: Fixed width value files. This enables MetaModel to read flat files where every value has the ... [More] same length (ie. not separated as such, but formatted in character columns). Full support for paging queries. The query interface now has a setFirstRow(int) method, which in combination with the existing setMaxRows(int) method allows for paging and finer grained control over the resulting data sets. Bugfixes pertaining to DB2 support: Added specific dialect support to ensure that queries for DB2 are formatted correctly, especially with regard to fully qualified schema names in queries. You can get the new MetaModel at our  downloads page or through the central Maven repositories: <dependency> <groupId>org.eobjects.metamodel</groupId> <artifactId>MetaModel-full</artifactId> <version>1.6</version> </dependency> We hope you enjoy MetaModel - please let us know and provide feedback at the forums. [Less]

Eobjects.org and its contributors are pleased to announce that  DataCleaner 2.0.2 has just been released. DataCleaner 2.0.2 is a minor, but not unimportant, release containing a few bugfixes and a set of 8 feature enhancements: Tabs and buttons ... [More] in the workbench are disabled when no source columns have been selected. A special widget have been added to the "Source" tab, making it very easy to apply row count based sampling of the input data. When possible, filters now have the ability to optimize the query of a job (aka. Push-down optimization). This was implemented for the "Max rows", "Equals" and "Not null" filters. The growing amount of transformers caused a long list in the "Add transformer" popup. Therefore transformers are now grouped by category and displayed accordingly. The visualization of execution flow now allows removing column items and filter outcome items, making the graph more comprehensible, especially for very large jobs. The "Coalesce string" transformer now has a "Consider empty strings as null" flag, which is particularly useful when dealing with CSV files. Text-based dictionaries and synonym catalogs will get their cached values flushed, if the file they read from changes. The "Convert to date" transformer now includes the ability to specify your own date masks, if date strings require it. A bug was fixed when passing null values to the the email standardizer. A bug was fixed pertaining to proper presentation of "mixed" tokens in the the Pattern finder. With these improvements in place we see that DataCleaner 2.x is really catching along and we're very pleased with the quality and pace of improvements we are seeing. Go to the  Downloads page right away to grab the new version. [Less]

Since the release of DataCleaner 2.0, we've seen a renewed interest and a lot of activity around eobjects.org, DataCleaner and Human Inference. We're happy to get all this valuable feedback and it has also meant that there where some low hanging ... [More] fruit to as well as a few very minor bugs that we could easily add into the existing DataCleaner 2.0 release. This is why, already a week after 2.0 was released, we're releasing an update: 2.0.1. The update consist of minor updates: Filter outcomes where added to the flow visualization. A bug was fixed in the widget for selecting the tokenizer's separators. The "Equals" filter can now have multiple values to compare with. Some minor cosmetical improvements. For more detail, take a look at the milestone contents at Trac. DataCleaner 2.0.1 is available at the  downloads page and the update has also been automatically applied to our Java Web Start users. [Less]

The Open Source software community eobjects.org is happy to announce the release of DataCleaner 2.0. This release marks the biggest advance in technology and features for the DataCleaner platform throughout the history of the project. Amongst ... [More] exciting new features in DataCleaner 2.0 are: Data transformations, allowing you to preprocess, extract, refine, combine and calculate data items as a part of your data profiling jobs. Filtering, sampling and subflow management, allowing you to define criteria to exclude and include particular items of data. Richer reporting with charts, graphs, navigation trees and more. A bunch of new data quality functions for date gap analysis, phonetic similarity finding, synonym lookups and more. More configuration options and added data quality measures for existing data quality functions like the Pattern finder, String analyzer and more. Reusable profiling jobs, where you define your processing flow once and consequently run it on any data. Support for MS Excel 2007+ spreadsheets. For more information about what’s new in DataCleaner 2.0, see the  full list of new features in DataCleaner 2.0. Today it was also announced that  Human Inference, the European data quality authority has finished their acquisition of the eobjects.org site, to actively enter the market for entry-level Open Source data quality products. All projects on eobjects.org will remain open source and the benefit for the community and the products are apparent. The release of DataCleaner 2.0 is the first visible outcome of the acquisition, resulting from several months of intense cooperation between Human Inference and the community members, to put together a state-of-the-art data profiling application. For more information about the eobjects.org acquisition, see the  press release on the Human Inference website. Times are really exciting in the eobjects.org community these days. We hope you’re all as enthusiastic about the new DataCleaner 2.0 as we are. The application is ready for download and for immediate launch through Java Web Start, so visit the  DataCleaner website now. [Less]

MetaModel 1.5, an Open Source Java framework for accessing, exploring and querying different datastores using a unified API, have just been released.  MetaModel provides a single view and a SQL/LINQ-like query engine for everything ranging from ... [More] relational databases, CSV files, Excel spreadsheets, XML files, dBase (.dbf), MS Access (.mdb) and OpenOffice.org (.odb) databases. The 1.5 release has been more than a year under way, including substantial new features and enhancements. Three major themes influence the new features of the 1.5 release: Improved datastore compliancy In addition to the already extensive set of supported datastore types, the following new datastore features have been added: Support for Excel 2007+ (.xlsx) spreadsheets has been added. Composite datastores have been added, allowing you to define queries that span multiple datastores. Excel formula calculation have been added. Fluent Query Builder API MetaModel 1.5 retains the existing Querying API, which is extremely flexible but also complex, and therefore quite easy to make mistakes with. But MetaModel 1.5 adds a new layer of abstraction to the Querying API: The Query Builder API. With the Query Builder API you can define queries in an even easier, more safe and elegant way. The goal of the Query Builder API is to leverage the use of the compiler as far as possible for query expression. An example demonstrates it quite well: DataContext dc = DataContextFactory.create[your_datastore_type]DataContext (...); Query q = dc.query() .'''from'''(projects).'''selectCount'''().'''and'''(community) .'''where'''(license).'''equals'''("oss") .'''groupBy'''(community).toQuery(); Interfaces and immutability Instead of the previous JavaBeans based API, the 1.5 release includes interfaces for just about everything in the library. This means that it is as of now easier to test, integrate and deploy MetaModel. It also allows for better encapsulation internally as well as improved safety by exposing only immutable variants of the data structures (like Table, Schema, Column etc.) that are modifiable only by the framework. Today it was also announced that  Human Inference, the European data quality authority has finished their acquisition of the eobjects.org site, to actively enter the market for entry-level Open Source data oriented applications. All projects on eobjects.org, including MetaModel, will remain Open Source, but heavily enforced by the invested time and resources that Human Inference is adding to these projects. For more information about the eobjects.org acquisition, see the  press release on the Human Inference website. MetaModel is already in use in a lot of projects, including the  DataCleaner data analysis/profiling application and  Quipu, the data warehouse generator. It is also in Human Inference’s plans to expand the usage of MetaModel into their enterprise-grade data matching and deduplication applications. If you think MetaModel 1.5 sounds interesting,  head over to the website to learn more. MetaModel is available as a  Maven artifact or as a traditional  download at Google code. [Less]