What is new in MongoChef 4.5

We are immensely excited to announce the release of MongoChef 4.5!

This is one of our largest releases ever. We would like to explicitly thank you, our users, for your plentiful feedback and feature suggestions that continue to make MongoChef so much better.

MongoChef 4.5 is chock-full of new features and improvements:

  • Full MongoDB 3.4 Compatibility
  • MongoDB Export to SQL
  • Improved Connections Tree
  • Integration of mongodump and mongorestore for Easy Backups

to name but the most high-level ones.

Full MongoDB 3.4 Compatibility

MongoDB 3.4 has brought many new high-level features such as read-only views, collation documents, and new aggregation operators. MongoChef 4.5 adds full support for these features.

Read-Only Views

MongoChef makes it super-easy to create MongoDB read-only views with its industry-leading aggregation builder. Build and debug your view with ease and then store it as a read-only view with just a click of your mouse.

mongodb create view

Collation Support

With MongoDB 3.4, you can now provide a collation document to specify language-specific rules for string comparison, such as rules for letter case and accent marks. MongoChef 4.5 allows you to specify a collation when you’re creating a collection, view, or an index and also when building an aggregation pipeline – and, naturally, wherever applicable in IntelliShell.

mongodb collation support

New Aggregation Operators

MongoDB 3.4 has also brought a number of new aggregation operators ($bucket, $bucketAuto, $facet, $graphLookup, $sortByCount). Of course, both MongoChef’s aggregation builder and IntelliShell have you covered here.

Export Your MongoDB Data to SQL

MongoChef Pro and Enterprise now let you export your MongoDB data directly to SQL and support the major SQL dialects: Oracle, MS SQL Server, MySQL, and PostgreSQL. Select your MongoDB data source and export to a SQL file that can then be safely imported into your SQL deployment.

1) Choose your SQL format options:

mongodb sql export options

2) MongoChef runs schema discovery on your MongoDB data to auto-select the most appropriate SQL target data types. You can of course also manually overwrite the mapping.

mongodb sql export mapping

3) View the resulting SQL file and safely import your MongoDB data into your SQL deployment.

mongodb sql file

Improved Connection Tree

We have introduced new meta-levels below the database level to make it even more comfortable to manage your various collections, views, system collections, and GridFS buckets.

connection tree metalevels

 

Integration of mongodump and mongorestore for Easy Backups

MongoChef now integrates with the mongodump and mongorestore backup and restore tools. You can now directly export (and of course import) your databases, collections or your query results to BSON folders or BSON archive files directly from within MongoChef. Backing up your data with MongoChef has never been easier.

1) Choose mongodump as export format:

export format mongodump

2) Configure your mongodump options:

mongodump configuration

Wait, there’s more!

See our full change log http://downloads.3t.io/mongochef-pro/changelog.txt for all the details.

 

What is new in MongoChef 4.4

MongoChef has just gotten even better! We are very pleased to announce that MongoChef 4.4 has been released!

For this release, we have particularly focussed on improving the overall user experience of MongoChef even further. Thanks a lot for all those suggestions!

Undocking tabs

Loads of you have requested this. And yes, this is a great productivity booster! You can now easily undock (and re-dock) any tab by just dragging the tab and dropping it off in a new location. Undocking tabs is especially useful in situations where you e.g. want to compare results or move a certain tab to a second monitor.

Filter connections

Over time, your connection list will likely grow to a substantial size. In order to quickly find a certain connection, we have added a filter box to the connection manager. As you type, it will show only those connections that match your input. MongoChef will search all relevant connection fields (name, database, connection type, etc):

mongochef connection filter

Read-only JSON document viewer

This is a seemingly minor addition to MongoChef – after all MongoChef has always had a full-blown JSON document editor. Nonetheless, this a very useful new feature, and an oft-requested one, as it will prevent you from inadvertently making changes to a document when just viewing it.

mongodb-json-document-viewer

Dark editor theme

We have implemented another very popular feature request: make the background of the editors in MongoChef dark. And, so, here it is (you can set it in the Preferences):

mongochef dark editor theme

We will be further fine-tuning this over the next iterations. So, please send us your feedback :-)

Import empty CSV fields as empty strings

We have added an option to our CSV import. You can now specify to import empty CSV fields as empty strings.

mongodb csv import empty strings

 

Wait, there’s more!

See our full change log http://downloads.3t.io/mongochef-pro/changelog.txt for all the details.

 

How to rename a field in MongoDB using MongoChef

Every now and then you may need to change the name of a field within your MongoDB documents. Whether you need to do it for a single document, or perhaps for multiple documents, this task can be easily accomplished with only a few steps from within the MongoChef app.

Renaming A Field Name In A Single Document Within MongoDB

The simplest of these tasks is changing a field name in single document. See the steps below:

  1. Open the document in which the field to be renamed is located
  2. Double click on the field name (In this case “dob”)
  3. You will now have the option to type the new field name into this space
  4. Once you press “Enter” the new field name will be saved

MongoChef Support, MongoChef, MongoDB

Renaming A Field Name Within Documents Matching A Query

Should you wish to change all documents matching a particular query then this can also be easily achieved in just a few steps. In this example we are changing the “dob” field in all documents relating to customers with the last name “Miller”. See below:

  1. Type in your query: {last:"Miller"} – and hit “Enter”
  2. Open one of the documents resulting from the query search
  3. Find the field that you would like to rename (“dob”) and right click it.
  4. Select “Rename Field”
  5. Type in the new field name (“date_of_birth”) and hit“Enter”
  6. Select “Documents matching query criteria” from the dropdown box
  7. Click “Enter”
  8. Now every document within this query will have the field name “date_of_birth” in place of “dob”

MongoChef Support, MongoChef, MongoDB

Renaming A Field Name In All Documents Within A Collection

Finally, to rename a field in all documents within a collection follow the steps below (identical to the steps above apart from no. 4):

  1. Find the field that you would like to rename (again “dob”) and right click it
  2. Select “Rename Field”
  3. Type in the new field name (“date_of_birth”) and hit “Enter”
  4. Select “All documents in collection” from the dropdown box
  5. Click “Rename”
  6. Now every document in the entire collection will have the field name “date_of_birth” in place of “dob”

MongoChef Support, MongoChef, MongoDB

 

That’s it for this post, but if you still have a little time, please check out the other very useful features of MongoChef!

What is new in MongoChef 4.3.0

We have just released MongoChef 4.3.0!

This release brings many improvements and new features you have asked for, for example:

Schema Documentation

You can now generate beautiful schema documentation for MS Word:

mobgodb schema documentation

 

mongodb schema documentation word overview

 

mongodb schema documentation word

 

Explore documents missing a certain field

When you spot a field in the schema explorer that does not occur in all documents, you can now easily explore the documents that do not have that field.

mongodb schema discovery explore missing field

 

Wait, there’s more!

See our full change log http://downloads.3t.io/mongochef-pro/changelog.txt for all the details.

 

Real-Time Server Status Charts for MongoDB in MongoChef

This blog post demonstrates the Server Status Charts feature of MongoChef for MongoDB.

The server status charts show real-time updates of what’s happening on a MongoDB instance:
server-status-charts-allso you can easily monitor particular things of interest on production, development, test, or local instances.

You can open a Server Status Charts tab by right-clicking on a connection (or database, or collection) and select ‘Server Status Charts’ from the ‘Show Server Info’ menu.
server-status-charts-context-menu
You can customise which charts are shown, by clicking on ‘Select Charts’ in the upper right hand side corner of the Server Status Charts tab:
select-server-status-chartsFor example, if we wanted to focus on Operation Counts, Current Connections and Current Network Traffic, we would select only those, and the charts will be laid out intelligently to accommodate them:
selected-charts

What the chart information means

In this section we’ll explain the information shown by each of the different charts.

Note that the information displayed in the charts may be:

  • Cumulative – where the charts show information since the start of a particular point until the present moment, e.g. all events of a particular type since the database started.
  • Per Update Period – where the charts show information about a particular event that occurred during an update period, e.g. if we have the ‘Update frequency’ set to ‘2 sec’, then the chart will show all events of that particular type that occurred within 2 second periods.
  • Point in Time – where the charts show information about a particular event as it was at the point in time that the server status was requested from the database.

The table below details each of the server status charts:

Operation Counts
Per Update Period
The number of database operations by type during the update period.
Active Clients
Point in Time
The number of the active client connections performing read and write operations.
Current Queue
Point in Time
The number of operations that are currently queued and waiting for the read or write lock.
Current Connections
Point in Time / Per Update Period
The number of incoming connections from clients to the database server. ‘In use’ and ‘Available’ report the point in time values, ‘Total created’ shows values per update period.
Current Network Requests
Per Update Period
The number of distinct requests that the server has received during the update period.
Current Network Traffic
Per Update Period
The number of bytes that reflects the amount of network traffic received by and sent from this database during the update period.
Total Connections Created
Cumulative
Count of all incoming connections created to the server. This number includes connections that have since closed.
Total Network Requests
Cumulative
The total number of distinct requests that the server has received.
Total Network Traffic
Cumulative
The total number of bytes that reflects the amount of network traffic received by and sent from this database.

More detailed information about the counters the chart data are based upon can be found in the MongoDB documentation.

That’s it for this post, but if you still have a little time, please check out the other very useful features of MongoChef!

MongoDB Schema Discovery and Exploration with MongoChef

One of the great things that we love about MongoDB is of course that it is schema-less. This makes adapting your application to changing requirements a breeze. That said, your data will often have a fixed implicit schema – e.g. each document in your employees collection will likely always have a first and last name field. So, making sure from time to time that all documents indeed contain certain fields is probably a good idea. Likewise, having the power to dynamically add new or discontinue old fields in your documents can often lead to a proliferation of various “versions” of your document schema. So, getting a feel for how often a certain field occurs in your collection can be quite helpful.

Luckily, MongoChef Pro makes discovering and exploring the schema in your MongoDB collections super-easy! With MongoChef, you can quickly:

  • check the health of your schema
  • find schema anomalies
  • inspect data outliers
  • visualize data distributions

So, let’s dive right in!

Schema Discovery

Select the collection whose schema you want to explore (in this example “customers”) and click the “Schema” icon in the global toolbar. This will open a tab where you can now configure your schema discovery:

mongodb schema discovery

  1. Here you can control how MongoChef should sample documents from the collection for the schema discovery. By default MongoChef will analyze randomly selected documents. You can choose between “Random”, “First”, “Last”, or “All” – in which case MongoChef will read in all documents in the collection.
  2. MongoChef gives you full control over how many documents should be read for the schema discovery. For our example, we will look at 1,000 documents.
  3. By default, MongoChef will not analyse the elements of any array fields when it encounters them. The reason is that arrays can often contain thousands of elements of the same type which can lead to bloated schema results. You can of course override this default behavior.
  4. You can also provide a query to further control the document set that should be used for the schema discovery. In our example, we will use an empty query, which will return all documents in the collections
  5. Click “Run analysis” to start.

After the analysis has completed, you will see the schema discovery result page:

mongodb schema discovery result

In the left-hand pane (1), you will see the discovered schema tree. For each field you see its name, its global probability – i.e. which percentage of documents were found to contain that field, and its discovered field type(s). You can now easily explore your schema as you would with a JSON document.

In the right-hand pane (2), MongoChef displays information about the type and data distribution of the currently selected schema field.

Verifying Your Schema

The schema tree is a great tool to understand and verify your schema. For our example, we assume a – fictitious – “customers” collection which contains the personal information of the customers of our – equally fictitious – shop. With the schema tree, we can now easily verify that required fields like “name” and “transactions” do indeed occur in 100% of our documents. We can also observe that the optional field “title” is apparently provided by 56.7% of our customers. The schema tree is also really helpful in discovering schema anomalies:

Discovering Missing Fields

mongodb schema discovery missing fields

Looking at our schema discovery results, we see that field “first” – which stores our customers’ first names – is missing in 0.4% of our documents. This may for example suggest that our web shop software might be flawed. To learn more about those documents that are missing a “first” field, MongoChef makes it super-easy to explore that actual data. If we right-click the “first” field, we can choose “Explore documents not containing selected field”. This will open a new query tab that shows all documents that do not contain the field “first”. It is important to note that this query will of course honor the base query criteria of your analysis but will bring up all matching documents in the collection, not just in the (limited) sample set.

mongodb schema discovery inspect missing fields

We could now inspect those documents which might reveal clues as to what has caused the missing field anomaly.

Discovering Unexpected Fields

Looking at our schema discovery results again, we spot another anomaly – an unexpected field:

mongodb schema discovery unexpected field

We see that in 95% of our documents, the field “user_name” is spelled with an underscore. Yet, in 5% of documents, it is spelled with a hyphen. This could for example be down to a typo somewhere in the source code. After fixing the typo in the code, MongoChef then makes it very easy to also fix it in your collection. Right-click the incorrectly-spelled “user-name” field and select “Explore documents containing selected field”.

mongodb-schema-discovery-explore-documents

This will open a new query tab showing all documents containing the selected field (“user-name”). Here right-click the field to rename it:

mongodb schema discovery rename field

and choose “All documents in collection” in the ensuing dialog to rename all occurrences in the collection:

mongodb rename all

Easy-peasy :-)

Discovering Incorrect Field Types

Another type of schema outlier that one commonly wants to look out for is incorrect field types. MongoChef makes that really easy to spot too. Consider in our example the “address” field in our customer documents. We store the addresses as an embedded object of the following type:

“address”:{“street”:{“name”:”Scofield”,”suffix”:”Drive”,”number”:”34″}}

However, when we look at the “address.street” field in the discovered schema tree, we can see that there appear to be some outliers where “address.street” is of type String:

mongodb schema discovery outlier

When we select the String instance of “address.street” in the schema tree, we can quickly see in the right-hand data pane that in 4 documents something must have gone awry and all street information was stored in a simple string.

mongodb schema discovery type outlier

We can right-click the String instance node in the schema tree to “Explore documents with selected field of type String” to have a closer look at those documents:

mongodb schema discovery explore documents

 

mongodb schema discovery document view

Exploring Data Distributions

As we have seen, when you click a field or one of its type instances in the schema tree, you can see charts showing – depending on the selected data type – various statistics on the type or data distribution of the field. Now, while MongoChef is not a full-blown BI tool by any means, these data distribution charts can often already give you some very useful insights into your documents.

Value Histograms

For numeric fields, one of the stats charts that the right-hand pane shows is the value histogram. If we look for example at the value histogram of our “transactions” field, we can quickly observe that most customers have engaged in around 50 transactions, with tails of less common numbers of transactions.

mongodb schema discovery histogram

Top Values

For many data types, the right-hand panel also shows the top values that were found across the analyzed documents. This can often be helpful to spot data outliers. Consider the “package” field in our example. Suppose our customers can subscribe to a “Free”, “Basic”, “Standard”, “XL”, or “XXL” package. However, we see that there are some customers who seem to have a “Beginner” package – which may indicate a backend glitch, for example.

mongodb schema discovery top values

Luckily, we can now use MongoChef to edit those values directly in-place:

mongodb schema discovery in place editing

Date Distributions

For date fields, MongoChef shows you in detail the value distributions. Consider, in our example, the “registered_on” field. If we look at its “Monthly value distribution”, we notice that customer registration seems to be particularly strong in the summer as well as in January. This might then provide valuable feedback to the marketing and sales teams.

mongodb schema discovery date distribution

Summary

MongoChef provides a very powerful schema explorer feature that lets you easily discover the schema that is present across the documents in your collection and thereby helps you find schema and data outliers. Drilling down into individual fields, you can see, for each field, various visual statistics relevant to the data type of that field.

MongoDB schema discovery and exploration has never been easier!

What is new in MongoChef 4.2.0

Today we are very excited to announce that we have just released MongoChef 4.2.0!

This release brings many improvements and new features you have asked for, for example:

Improved SSH/SSL support

We have significantly expanded our SSH/SSL support. We now support most ciphers, key exchange protocols, message authentication codes (MACs) as well as DER-encoded private keys.

Option to remove a field from matching documents

When you select a field in document and you choose to remove the field, you can now also specify in the ensuing dialog to remove the field only from those fields matching the current query.
remove field from matching documents

Parse “connect-to” database from connection URI

When connecting to the mongo shell, one can pass the shell a connection URI in the following form

mongodb://user:pwd@host:port/my_db?authSource=admin

which will connect the user (who is defined in database “admin”) directly to the database “my_db”. This concept of course doesn’t directly translate to MongoChef where you see all available databases at once. MongoChef will now parse that connect-to database directly from the URI and add it to the list of manually defined visible databases for the user:

mongodb connection uri

mongodb manual database list

Save binary blob field to file

We have added an option directly to a binary field’s pop-up menu to save the binary blob to file (and to load from a file into the field).

save mongodb binary field to file

Scan entire collection for all fields for CSV export

When exporting documents to CSV, MongoChef by default scans only a subset of documents (usually the documents currently showing in the query result) for field names to suggest as CSV column names. Due to the schema-less nature of MongoDB, this could miss desired fields that in the past had to then be added manually. Now, MongoChef offers a new option to force a scan of the entire collection – which will continuously update the field name list as they are discovered.

Discover all fields for CSV export

Quick feedback when two compared collections are identical

We have added quick and obvious visual feedback to show the user in case the source and target collection during a comparison analysis were found to be identical.

Compare MongoDB collections - identical

Wait, there’s more!

See our full change log http://downloads.3t.io/mongochef-pro/changelog.txt for all the details.

 

How to temporarily enable/disable stages in your MongoDB aggregation pipeline

With the release of MongoChef 4.1.1, we have a made it super-easy to quickly disable stages temporarily in your aggregation query. This often comes in very handy while debugging your aggregation query.

To demonstrate this, let’s consider the following simple aggregation query on the City of Chicago’s publicly available housing database.

debug aggregation query

Our aggregation pipeline consists of 4 stages:

  1. Limit the documents that go into the pipeline to just 5
  2. A projection stage that just keeps the address, zip and type fields
  3. A match stage that only keeps documents that have “60644” as zip code
  4. A sort stage that sort the documents in the pipeline based on their (property) type

While designing and debugging my aggregation query, we limited my pipeline to just 5 documents to keep the load on the server as low as possible. Now that we think our query is done, we want to test it for the first time on the entire data set. Since we are not yet entirely convinced that the query does what it is supposed to, we don’t want to get rid of the limit stage just yet. Instead, we’ll just temporarily exclude it from the pipeline. For this, we simply click the stage tab and deselect the “Include in pipeline” option:

disable aggregation stage

When we now run the aggregation query, it will be executed without the (disabled) first stage and we can see that the full result set is returned:

full aggregation result

And that’s how easy it is to temporarily disable aggregation stages with MongoChef! And here once again in one fluid motion :-) .

disable aggregation stage

 

How to Color Your Connections and Databases in MongoChef

MongoChef now has a coloring feature that is aimed at helping you quickly differentiate visually between your live data and test data and avoid possible accidents coming from mixing these two. This is a short walk through on how to use this useful little feature.

Let’s face it, we have all been there. Or, it is actually not that unlikely that, in case you have not yet made this mistake, you will, at some point, do the unthinkable and confuse your test data and live data. If you are religious about avoiding this mistake, you are probably already applying a number of personal tricks to make sure you don’t fall into this trap.

MongoChef makes your life easier. With just one right click of your mouse, you can assign for example a red color to your live database

Coloring DB
and a green color to your test database. (I have made a local copy of my live database for this purpose, which I will consider my test database)

Coloring test DB

Now, you can have both databases open at the same time and keep switching back and forth between them while you play with your data. Your red tabs will now immediately tell you that you are working on your live database and that you should pay extra attention when you run update queries on them.

Coloring test and live DB
All nice and clean in one glance! Saves you time and trouble and now you can rest assured you will never make the mistake of updating your live database when thinking you were in fact working on your test one!
You can even play with horizontal and vertical split of tabs for a quick and easy glance at pairs of data sets.

How to Have a Side-By-Side View of Multiple Tabs with MongoChef

This blog post is a visual walk through of how MongoChef has implemented one of your most requested features which is the ability to have side-by-side tabs.

Very frequently, you probably encounter one of these use cases:

  1. You want to easily compare test and production data sets
  2. You simply want to look at and quickly compare result sets of different queries.

For the purposes of this blog post, let’s focus on the second use case and look at two separate views of the same collection in the collection view. This will result in two different tabs displayed one after the other like this:

Split Tabs Vertically

Hmm, this is not the optimal view when you want to see and compare data sets, is it?

View data side-by-side

With the new Split Vertically / Split Horizontally feature that we have just implemented, MongoChef allows you to quickly see, in a single glance, multiple tabs side by side.

You can choose to display the two data-sets side-by-side vertically or horizontally. Start by right clicking on any of the tabs and choose Split Vertically

Split Tabs Vertically

Now your two tabs will be shown side-by-side like this:

Split Tabs Vertically

Or you could choose to split your view horizontally. Again, start by right clicking one of your tabs and now choose Split Horizontally.

Wait, there’s more!

You can now even drag-and-drop tabs and reorder them in case you have multiple tabs open and you want a quick way to compare them more easily.

Drag and drop tabs
While you continue to use MongoChef, take some time to check out the neat list of feature requests from your fellow MongoChef colleagues. You can vote on the features which matter most to you, or propose new features that have not yet been requested by anyone. After all, we are here to make your life and MongoChef better.