Datumbox Machine Studying Framework model 0.8.0 launched

Datumbox Framework v0.8.0 is out and packs a number of highly effective options! This model brings new Preprocessing, Characteristic Choice and Mannequin Choice algorithms, new highly effective Storage Engines that give higher management on how the Fashions and the Dataframes are saved/loaded, a number of pre-trained Machine Studying fashions and many reminiscence & velocity enhancements. Obtain it now from Github or Maven Central Repository.

One of many primary targets of model 0.8.0 was to enhance the Storage mechanisms of the framework and make disk-based coaching accessible to all of the supported algorithms. The brand new storage engines give higher management over how and when the fashions are being endured. One vital change is that the fashions should not being saved routinely after the match() methodology is completed however as a substitute one must explicitly name the save() methodology offering the identify of the mannequin. This allows us not solely to discard simpler non permanent algorithms with out going by means of a serialization part but additionally to avoid wasting/load the Dataframes:

Configuration configuration = Configuration.getConfiguration();
Dataframe information = ...; //load a dataframe right here

MaximumEntropy.TrainingParameters params = new MaximumEntropy.TrainingParameters();
MaximumEntropy mannequin = MLBuilder.create(params, getConfiguration());
mannequin.match(information);
mannequin.save("MyModel"); //save the mannequin utilizing the precise identify
mannequin.shut();

information.save("MyData"); //save the information utilizing a selected identify
information.shut();

information = Dataframe.Builder.load("MyData", configuration); //load the information
mannequin = MLBuilder.load(MaximumEntropy.class, "MyModel", configuration); //load the mannequin
mannequin.predict(information);
mannequin.delete(); //delete the mannequin

Presently we assist two storage engines: The InMemory engine which could be very quick because it hundreds the whole lot in reminiscence and the MapDB engine which is slower however permits disk-based coaching. You’ll be able to management which engine you utilize by altering your datumbox.configuration.properties or you possibly can programmatically modify the configuration objects. Every engine has its personal configuration file however once more you possibly can modify the whole lot programmatically:

Configuration configuration = Configuration.getConfiguration(); //conf from properties file

configuration.setStorageConfiguration(new InMemoryConfiguration()); //use In-Reminiscence engine
//configuration.setStorageConfiguration(new MapDBConfiguration()); //use MapDB engine

Please word that in each engines, there’s a listing setting which controls the place the fashions are being saved (inMemoryConfiguration.listing and mapDBConfiguration.listing properties in config recordsdata). Be sure to change them or else the fashions shall be written on the non permanent folder of your system. For extra data on the way you construction the configuration recordsdata take a look on the Code Instance venture.

With the brand new Storage mechanism in place, it’s now possible to share publicly pre-trained fashions that cowl the areas of Sentiment Evaluation, Spam Detection, Language Detection, Subject Classification and all the opposite fashions which can be accessible by way of the Datumbox API. Now you can obtain and use all of the pre-trained fashions in your venture with out requiring calling the API and with out being restricted by the variety of every day calls. Presently the revealed fashions are skilled utilizing the InMemory storage engine and so they assist solely English. On future releases, I plan to supply assist for extra languages.

Within the new framework, there are a number of modifications on the general public strategies of most of the courses (therefore it isn’t backwards suitable). Essentially the most notable distinction is on the way in which the fashions are initialized. As we noticed within the earlier code instance, the fashions should not straight instantiated however as a substitute the MLBuilder class is used to both create or load a mannequin. The coaching parameters are supplied on to the builder and so they can’t be modified with a setter.

One other enchancment is on the way in which we carry out Mannequin Choice. The v0.8.0 introduces the brand new modelselection package deal which affords all the required instruments for validating and measuring the efficiency of our fashions. Within the metrics subpackage we offer an important validation metrics for classification, clustering, regression and suggestion. Word that the ValidationMetrics are faraway from every particular person algorithm and they’re now not saved along with the mannequin. The framework affords the brand new splitters subpackage which permits splitting the unique dataset utilizing completely different schemes. Presently Ok-fold splits are carried out utilizing the KFoldSplitter class whereas partitioning the dataset right into a coaching and take a look at set might be achieved with the ShuffleSplitter. Lastly to shortly validate a mannequin, the framework affords the Validator class. Right here is how one can carry out Ok-fold cross validation inside a few strains of code:

ClassificationMetrics vm = new Validator<>(ClassificationMetrics.class, configuration)
    .validate(new KFoldSplitter(ok).cut up(information), new MaximumEntropy.TrainingParameters());

The brand new Preprocessing package deal replaces the previous Knowledge Transformers and offers higher management on how we scale and encode the information earlier than the machine studying algorithms. The next algorithms are supported for scaling numerical variables: MinMaxScaler, StandardScaler, MaxAbsScaler and BinaryScaler. For encoding categorical variables into booleans you need to use the next strategies: OneHotEncoder and CornerConstraintsEncoder. Right here is how you need to use the brand new algorithms:

StandardScaler numericalScaler = MLBuilder.create(
    new StandardScaler.TrainingParameters(), 
    configuration
);
numericalScaler.fit_transform(trainingData);

CornerConstraintsEncoder categoricalEncoder = MLBuilder.create(
    new CornerConstraintsEncoder.TrainingParameters(), 
    configuration
);
categoricalEncoder.fit_transform(trainingData);

One other vital replace is the truth that the Characteristic Choice package deal was rewritten. Presently all characteristic choice algorithms give attention to particular datatypes, making it doable to chain completely different strategies collectively. Consequently the TextClassifier and the Modeler courses obtain a listing of characteristic selector parameters relatively than only one.

As talked about earlier all of the algorithms now assist disk-based coaching, together with those who use Matrices (solely exception is the Help Vector Machines). The brand new storage engine mechanism even makes it doable to configure some algorithms or dataframes to be saved in reminiscence whereas others on disk. A number of velocity enhancements had been launched primarily because of the new storage engine mechanism but additionally because of the tuning of particular person algorithms corresponding to those within the DPMM household.

Final however not least the brand new model updates all of the dependencies to their newest variations and removes a few of them such because the the commons-lang and lp_solve. The commons-lang, which was used for HTML parsing, is changed with a quicker customized HTMLParser implementation. The lp_solve is changed with a pure Java simplex solver which signifies that Datumbox now not requires particular system libraries put in on the working system. Furthermore lp_solve needed to go as a result of it makes use of LGPLv2 which isn’t suitable with the Apache 2.0 license.

The model 0.8.0 brings a number of extra new options and enhancements on the framework. For an in depth view of the modifications please examine the Changelog.

Don’t overlook to clone the code of Datumbox Framework v0.8.0 from Github, try the Code Examples and obtain the pre-trained Machine Studying fashions from Datumbox Zoo. I’m trying ahead to your feedback and options.