• Differentiate between NoSQL and SQL databases.
• Explain what Mongoose is.
• Describe the role of Mongoose schema and models.
• Use Mongoose to perform CRUD functionality.
• List and describe common Mongoose queries.
• Persist data using Mongoose embedded documents.
• Describe how to use validations in Mongoose.

Before we dive into Mongoose, let's talk a bit about about Mongo/NoSQL.

A NoSQL database is a non-relational database.

• This means that there is not an explicit one-to-one, one-to-many and many-to-many relationship.
• That being said, we can emulate these relationships in a NoSQL database.

A NoSQL database are organized into documents and collections.

• Collections are the NoSQL equivalent of tables in a SQL database.
• Documents are the NoSQL equivalent of a table row.

This is not the only way that a NoSQL database organizes data. Check out this link to learn more.

MongoDB is a NoSQL database that stores information as JSON.

• Technically, it's BJSON or "binary JSON," pronounced "BSON". We will discuss BSON more in a few minutes.

It is flexible.

• You don't need to follow a schema if you don't want to. This might be helpful with non-uniform data.
• That being said, you can enforce consistency in your data by using schemas. In fact, we will be doing that in today's class. And I recommend that you follow this same work-flow.

It is fast.

• Data is "denormalized" in NoSQL, which means that it is all in the same place.
• For example, a post's comments will be nested directly within that post, in the database.
• This is the opposite of a relational database, in which we need to make queries in order to retrieve data in other tables. The tables are connected through a relation.

Many web applications already implement object-oriented Javascript.

• If we are using objects in both the back-end and front-end, it makes handling and sending data between the client and the database much easier.
• There is no need for type conversion (e.g., making sure a Ruby hash is being served as JSON).

Even though you won't be writing very many MongoDB commands in SEI, we will use some MongoDB CLI commands to test what is in our database. Examples include...

• show dbs - Will show a list of all databases
• use database-name - Will connect to a database
• show collections - Will list the collections that exist within a database
• db.students.find() - Will list all of the students in the student collection

• A MongoDB database consists of collections of documents.
• A document in MongoDB is composed of field (key) and value pairs.

Lets take a look of what a MongoDB document may look like:

{
    _id: ObjectId("5099803df3f4948bd2f98391"),
    name: { first: "Alan", last: "Turing" },
    birth: new Date('Jun 23, 1912'),
    death: new Date('Jun 07, 1954'),
    contribs: [ "Turing machine", "Turing test", "Turingery" ],
    views: 1250000
}

Who is Alan Turing?

What does this data structure remind you of? JSON!

A MongoDB document is very much like JSON, except that it is stored in the database, in a format known as BSON (think Binary JSON).

BSON basically extends JSON with additional data types, such as ObjectID and the Date shown above.

The _id is a special field that represents the document's primary key and will always be listed as the first field. It must be unique. Primary keys will become very important in relational/SQL databases, and also when we discuss embedded documents.

We can explicitly set the _id like this:

{
  _id: 2,
  name: "Suzy"
}

or this...

{
  _id: "ABC",
  name: "Suzy"
}

However, it is more common to allow MongoDB to create it implicitly for us, using it's ObjectID data type.

"Let's face it, writing MongoDB validation, casting and business logic boilerplate is a drag. That's why we wrote Mongoose."

Mongoose is an ODM (Object Data Mapping), that allows us to encapsulate and model our data in our applications. It gives us access to additional helpers, functions and queries to simply and easily perform CRUD actions.

Mongoose will provide us with the similar functionality to interact with MongoDB and Express, as Active Record will do with PostgreSQL and Rails.

Active Record is an ORM (Object Relational Mapper). An ODM is the same thing as an ORM, without the relations.

Here's an example of some Mongoose code pulled from their documentation.

let mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/test');

let Cat = mongoose.model('Cat', { name: String });

let kitty = new Cat({ name: 'Zelda' });

kitty.save(function (err) {
  if (err) // { do something }
  
  console.log('meow');
});

1. Create a new project folder in your /class-exercises folder: $ mkdir mongoose-intro-lesson

2. cd into the new folder.

3. $ npm init -y

4. Create a new database directory inside that folder: $ mkdir db

5. Create the following files: $ touch server.js db/schema.js db/seeds.js

6. Open the app in VSCode: $ code .

   • make sure that your entry point is server.js, and not index.js.

Your folder structure should look like this:

Walk through the previous steps to set-up your app

For today's in-class Mongoose demonstrations, we will be creating an app that uses two models: Students and Projects.

Let's begin by installing Mongoose into our project folder...

$ npm install mongoose

In order to have access to mongoose in our application, we need to explicitly require Mongoose and open a connection to the database that we specify.

// in the db/schema.js

let mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/students');

The name above students will be the name of the app's Mongo database. A cool feature of Mongo is that we don't have to create this database initially. If the database doesn't exist Mongo will instantiate it the first time you try to save something.

// in the db/schema.js

let mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/students' , { useNewUrlParser: true , useUnifiedTopology: true, 
useFindAndModify: false, useCreateIndex: true })

// Now that we are connected, let's save that connection to the database in a letiable. We are just doing this to keep our code DRY.
let db = mongoose.connection;

// Will log an error if db can't connect to MongoDB
db.on('error', function(err) {
  console.log(err);
});

// Will log "database has been connected" if it successfully connects.
db.once('open', function() {
  console.log("database has been connected!");
});

// Disconnect from database
db.close();

Now let's run our db/schema.js file:

$ node db/schema.js

Walk through the previous steps

• Everything in Mongoose starts with a Schema!
• Mongo is schema-less and doesn't require much structure (but it is still a good idea to use a schema).
• Schemas are used to define attributes and structure our documents.
• Each Schema maps to a MongoDB collection and defines the shape of the documents within that collection.

Here's an example of a Mongoose schema...

// in the db/schema.js

// First, we instantiate a namespace for our Schema constructor defined by mongoose.
let Schema = mongoose.Schema;

let StudentSchema = new Schema({
  name: String,
  age: Number
});

Mongo will add a primary key to each object using ObjectId. This will be referenced as an id property on each new instance of the object.

Mongoose Models will represent documents in our database.

• They are essentially constructor functions, which will allow us to preform CRUD actions within our MongoDB Database.
• Models are defined by passing a Schema instance to mongoose.model.

// db/schema.js

let Schema = mongoose.Schema;

// First, we instantiate a namespace for our Schema constructor defined by mongoose.
let StudentSchema = new Schema({
  name: String,
  age: Number
});

let StudentModel = mongoose.model("Student", StudentSchema);

.model() makes a copy of the schema.

• The first argument is the singular name of the collection that your model is for. Mongoose automatically looks for the plural version of your model name.
• This means that the model name Student will correspond to the students collection in the database.

Add the code above to your db/schema.js file

Mongoose is an ODM - an Object Document Mapper - i.e. it maps objects to documents. Therefore, Mongoose gives us the ability to perform the CRUD operations on a MongoDB database using JavaScript objects as our model objects.

A model object is an object whose primary concern is to hold data, and represent an instance of a Domain Model. For example, if we were developing an application for Delta, we might have Domain Models for Airports, Flights, Passengers, Luggage, etc. Instances of an Airport Domain Model might be ATL, LAX, JFK, ABQ, etc.

Let's add another model to our db/schema.js.

• We will be adding a schema for Project, since I want to create an application that tracks Students and Projects.
• Like a one-to-many relationship in a relational database, a Student will have many Projects.

In Mongoose, we create this relationship by using embedded documents.

Embedded Documents -- sometimes referred to as "sub-documents" -- are schemas of their own, as well as elements in a parent document's array

• They contain all of the same features as normal documents.

// in the db/schema.js

let ProjectSchema = new Schema({
  title: String,
  unit: String
});

let StudentSchema = new Schema({
  name: String,
  age: Number,
  projects: [ProjectSchema]
});

let ProjectModel = mongoose.model("Project", ProjectSchema);
let StudentModel = mongoose.model("Student", StudentSchema);

The projects key in your StudentSchema document, will contain a special array that has specific methods that work with embedded documents.

The Project Schema must be defined prior to our main Student Schema.

• Easy to conceptualize and set up.
• Can be accessed quickly.

• Do not scale well. Documents cannot exceed 16MB in size.

If you find that you are nesting documents within documents for 3+ levels, you should probably look into a relational database. Check Multiple collections (below) for more info.

// in the db/schema.js

let ProjectSchema = new Schema({
  title: String,
  unit: String,
  students: [{type: Schema.ObjectId, ref: "Student"}]
});

let StudentSchema = new Schema({
  name: String,
  age: Number,
  projects: [ {type: Schema.ObjectId, ref: "Project"}]
});

let StudentModel = mongoose.model("Student", StudentSchema);
let ProjectModel = mongoose.model("Project", ProjectSchema);

Set Up a Schema and the Model for Projects.

First let's create an instance of our Student model. Below is an example that describes one way of doing this...

// in the db/schema.js

// First we create a new student. It's just like generating a new instance of a constructor function!
let frankie = new StudentModel({ name: "Frankie P.", age: 30 });

// Then we save it to the database using .save

frankie.save()
  .then((student) => {
    console.log(student);
  })
  .catch((error) => {
    console.log(error)
  })

Wait, what is this .then and .catch thing all about? This is called a Promise! Let's take a few minutes to explore what this is:

app.get('/', (req, res) =>{
 //Do stuff when this route is hit on the server
})

$('div').on('click', function(e){
 //Do stuff when a user makes an interaction in the browser
})

fs.readFile('./data-csv', function(err, data){  // read a file
  if(err) throw err;

  csv.parse(data, function(err, csvData){       // parse the data as CSV
    if(err) throw err;

    var url = csvData[0].url;
    var req = http.request({host: url}, function(res){ // make an HTTP request
      var responseData = '';
      res.setEncoding('utf8');
      res.on('data', (chunk) => {
        responseData += chunk;
      });
      res.on('end', () => {                            // collect the response
        ResponseMessages.create({body: responseData},  // add record to a DB
          function(err, messageRecord){
          if (err) throw err;

          res.render('response', responseData);
        })
      });
    });
    req.on('error', (e) => {
      console.log(`problem with request: ${e.message}`);
    });

  })

});

//Disclaimer: getInfoFromDatabase is just an example method

getInfoFromDatabase()
  .then(function(data){ //Called when data is successfully fetched
    console.log("Successfully retrieved data");
    res.render('/', {
      data: data
    })
  })
  .catch(function(error){ //Gets called when data isn't successfully fetched
    console.error(error)

    res.send("Oh no, something went wrong")
  });

const axios = require("axios")

axios.get("http://www.omdbapi.com/?apikey=28cbce7c&s=starwars")
  .then((response) => {
    console.log("Success!")
    console.log(response.data)
  })
  .catch((error) => {
    console.log("Something went wrong")
    console.log(error.response.data)
  })

const axios = require("axios")

//Search for all movies that match Star Wars
axios.get("http://www.omdbapi.com/?apikey=28cbce7c&s=star%20wars")
  .then((response) => {
    //Get the ID of the first response and search for more info on that specific movie.
    console.log("Search Success!")
    console.log(response.data)

    const firstId = response.data['Search'][0].imdbID
    console.log("Id of the first movie", firstId);
    //Make a second API call to a third party API
    return axios.get(`http://www.omdbapi.com/?apikey=28cbce7c&i=${firstId}`)
  })
  .then((response) => {
    console.log("Second Api call success!");

    console.log(response.data)
  })
  .catch((error) => {
    console.log("Something went wrong")
    console.log(error.response.data)
  })

Run node db/schema.js in the Terminal to run it. We can also consolidate this code into a single .create method, like so...

// in the db/schema.js

StudentModel.create({ name: 'Frankie Q.', age: 31 })
  .then((student) => {
    console.log(student);
  })
  .catch((error) => {
    console.log(error)
  })

• Maybe we want to instantiate new instances of something for test purposes, but not actually save it to our database.
• Maybe we want to add something to the new object (like a date/timestamp) or do some additional validation checking before saving.
• SIDE NOTE: if you are using the new/save methods, we need to assign the object to a variable

• Create a new instance of a Student using new + .save
• Create a new instance of a Student using .create

Next, let's create a Project...

// db/schema.js

let anna = new StudentModel({ name: "Anna", age: 28 });
let project1 = new ProjectModel({ title: "memory game", unit: "JS" });

// Now we add that project to a student's collection / array of projects.
anna.projects.push(project1);

// In order to save that project to the student, we need to call `.save()` on the student -- not the project.
anna.save()
  .then((anna) => {
    console.log(anna)
  })
  .catch((error) => {

  })

Run node db/schema.js in the Terminal to run the code.

• Create a new Student and add a Project as an embedded document.

Let's delete this create code, and actually add some seed some data in our database, from our seeds.js file. In order to do that, we need to make sure that we can connect the schema.js to the seeds.js. Comment out the code we created above, and add the following to the bottom of the db/schema.js.

// in the db/schema.js

// The rest of our schema code is up here...

// By adding `module.exports`, we can reference these models in other files by requiring `schema.js`.
module.exports = {
  StudentModel: StudentModel,
  ProjectModel: ProjectModel
};

And add the following to db/seeds.js...

// in the db/seeds.js

let mongoose = require('mongoose');
let Schema = require("./schema.js");

let StudentModel = Schema.StudentModel;
let ProjectModel = Schema.ProjectModel;

Now let's call some methods in db/seeds.js that will populate our database...

// in the db/seeds.js

let mongoose = require('mongoose');
let Schema = require("./schema.js");

let StudentModel = Schema.StudentModel;
let ProjectModel = Schema.ProjectModel;

// First we clear the database of existing students and projects.
StudentModel.remove({})
  .then(() => {
    console.log('All students deleted!')
  })
  .catch((error) => {
    console.log(error)
  })

ProjectModel.remove({})
  .then(() => {
    console.log('All projects deleted!')
  })
  .catch((error) => {
    console.log(error)
  })

// Now, we will generate instances of a Student and of their Project.
const project1 = new ProjectModel({title: "Project 1!!!", unit: "JS"})
const project2 = new ProjectModel({title: "Project 2!!!", unit: "RAILS"})
const project3 = new ProjectModel({title: "Project 3!!!", unit: "REACT"})
const project4 = new ProjectModel({title: "Project 4!!!", unit: "EXPRESS"})

const projects = [project1, project2, project3, project4]

const ahmed = new StudentModel({name: "Ahmed" , projects: projects})
const maha = new StudentModel({name: "Maha", projects: projects})
const sami = new StudentModel({name: "Sami", projects: projects})
const salman = new StudentModel({name: "Salman", projects: projects})

const students = [ahmed, maha, sami, salman];

StudentModel.insertMany(students)
  .then(() => {
    console.log(`Added ${students.length} students to database.`)
  })
  .catch((error) => {
    console.log(error)
  })

Now, seed your database by running node db/seeds.js in your terminal. Use Ctrl + C to exit running Node.

Let's test if this all worked by opening Mongo in the Terminal...

$ mongo
$ show dbs
$ use students
$ show collections
$ db.students.find()

Add Seed data to your App

Oftentimes, when making a Mongoose query, we will pass in a callback function (or use a Promise with exec()). It will be passed two arguments: err and data.

• data contains the result of the Mongoose query.

Mongoose provides us with a variety of helper methods that allow us to easily retrieve documents from our database.

Explore them using the Mongoose Queries Documentation.

// Finds all documents of a specified model type. We can pass in a key-value pair(s) to narrow down the search.
Model.find({}, callback)

// Finds a single model by its id.
Model.findById(someId, callback)

// Find a single model using a key-value pair(s).
Model.findOne({someKey: someValue}, callback)

// Removes documents that match a key-value pair(s).
Model.remove({someKey: someValue}, callback)

Let's use .find() to implement the index functionality. We will do that in the controller file...

$ mkdir controllers
$ touch controllers/students_controller.js

We are adding a controllers directory and students_controller.js file to mimic how we might define a controller in an Express application. We will follow REST conventions and use our controllers to listen for incoming requests and communication with our database.

// in the controllers/students_controller.js

const express = require('express')
const router = express.Router()

const Schema = require("../db/schema.js")
const StudentModel = Schema.StudentModel
const ProjectModel = Schema.ProjectModel

router.get('/students', (request, response) => {
  StudentModel.find({})
    .then((students) => {
      response.render({  
        students: students
      })
    })
    .catch((error) => {
      console.log(error)
    })
})

module.exports = router

Now let's create a show method...

// in the controllers/students_controller.js

const express = require('express')
const router = express.Router()

const Schema = require("../db/schema.js")
const StudentModel = Schema.StudentModel
const ProjectModel = Schema.ProjectModel

router.get('/students', (request, response) => {
  StudentModel.find({})
    .then((students) => {
      response.send({ 
        students: students
      })
    })
    .catch((error) => {
      console.log(error)
    })
})

router.get('/students/:studentId', (request, response) => {
  const studentId = request.params.studentId

  StudentModel.findById(studentId)
    .then((student) => {
      response.send({
        student: student
      })
    })
})

module.exports = router

Follow the above instructions to implement index and show for the Student model.

Then use the Mongoose documentation to figure out how to Update and Delete students. If the documentation proves difficult to navigate, don't be afraid to Google it! We will go over how to Update and Delete after the exercise...

Don't look at these while working on the previous exercise!

// in the controllers/students_controller.js

const express = require('express')
const router = express.Router()

const Schema = require("../db/schema.js")
const StudentModel = Schema.StudentModel
const ProjectModel = Schema.ProjectModel

router.get('/students', (request, response) => {
  StudentModel.find({})
    .then((students) => {
      response.send({ 
        students: students
      })
    })
    .catch((error) => {
      console.log(error)
    })
})

router.get('/students/:studentId', (request, response) => {
  const studentId = request.params.studentId

  StudentModel.findById(studentId)
    .then((student) => {
      response.send({
        student: student
      })
    })
})


router.put('/students/:studentId', (request, response) => {
const studentId = request.params.studentId
const updatedStudent = request.body
StudentModel.findByIdAndUpdate(studentId, updatedStudent, {new: true})
  .then((student) => {
    response.redirect(`/students/${student._id}`)
  })
  .catch((error) => {
    console.log(error)
  })
})

module.exports = router

// in the controllers/students_controller.js

const express = require('express')
const router = express.Router()

const Schema = require("../db/schema.js")
const StudentModel = Schema.StudentModel
const ProjectModel = Schema.ProjectModel

router.get('/students', (request, response) => {
StudentModel.find({})
  .then((students) => {
    response.send({ 
      students: students
    })
  })
  .catch((error) => {
    console.log(error)
  })
})

router.get('/students/:studentId', (request, response) => {
const studentId = request.params.studentId

StudentModel.findById(studentId)
  .then((student) => {
    response.send({
      student: student
    })
  })
})

router.put('/students/:studentId', (request, response) => {
const studentId = request.params.studentId
const updatedStudent = request.body

StudentModel.findByIdAndUpdate(studentId, updatedStudent, {new: true})
  .then((student) => {
    response.redirect(`/students/${student._id}`)
  })
  .catch((error) => {
    console.log(error)
  })
})

router.delete('/students/:studentId', (request, response) => {
const studentId = request.params.studentId

StudentModel.findByIdAndRemove(studentId)
  .then(() => {
    response.redirect('/students')
  })
  .catch((error) => {
    console.log(error)
  })
})

module.exports = router

Adding the following to the server.js in order to access our routes after running npm install express body-parser --save

//in server.js

const express = require('express') //import package
const app = express() //execute the package
const bodyParser = require('body-parser') //used to parse the request body in JSON format
app.use(bodyParser.json()) //using the bodyparser in our application
const studentrouter = require('./controllers/students_controller.js') //importing our router
app.use('/', studentrouter) //using our router
app.listen(3000); // listening to post 3000 for incoming requests

Once done we can start the node server using node server.js

Lets use Postman to review our requests

Mongoose contains built-in validators and the option to create custom validators as well.

Validators are defined at the field level in a document, and are executed when the document is being saved. If a validation error occurs, the save operation is aborted and the error is passed to the callback.

Built in Validators:

• required and unique: used to validate the field's existence in Mongoose. These validators are placed in your schema, on the field that you want to validate.

Example: Let's say you want to verify the existence of a username field and ensure that it is unique before you save the user document.

let StudentSchema = new Schema({
  name: {
    type: String,
    unique: true,
    required: true
  }
  ...
});

This will validate the existence and uniqueness of the username field when saving the document, thus preventing the saving of any document that doesn't contain this field or the username already exists.

• match: type based validator for strings, added to the field's object, in your schema.

Continuing off the above example, to validate your email field, you would need to change your StudentSchema as follows:

let StudentSchema = new Schema({
  name: {
    type: String,
    unique: true,
    required: true
  },
  age: Number,
  email: {
    type: String,
    match: /.+\@.+\..+/
  }
});

The usage of a match validator here, will make sure that the email field value matches the given regex expression, thus preventing the saving of any document where the e-mail doesn't conform to a valid pattern!

• enum: helps to define a set of strings that are only available for that field value.

let StudentSchema = new Schema({
  name: {
    type: String,
    unique: true,
    required: true
  },
  age: Number,
  email: {
    type: String,
    match: /.+\@.+\..+/
  },
  role: {
    type: String,
    enum: ['Admin', 'Instructor', 'Student']
  },

});

By Adding an enum, we are adding a validation to ensure that only these three possible strings ('Admin', 'Instructor', 'Student') are saved in the document.

We can also define our own validators by using the validate property.

This validate property value is typically an array consisting of a validation function.

For example, say that we want to validate the length of a user's password. To do so, you would have to make these changes in your StudentSchema:

let StudentSchema = new Schema({
  ...
  password: {
    type: String,
    validate: [
      function(password) {
        return password.length >= 6;
      },
      'Password needs to be at least 6 characters.'
    ]
  },
});

• How is Mongoose used to interact with MongoDB?
• What are embedded documents in Mongoose?
• Why do we create a Schema in Mongoose?
• What do we need after Mongoose queries?
• What are some common built in validations for Mongoose? Why would we use them?

After this class you should be able to complete Part I of YUM.

• Mongoose Documentation
• Embedded Docs versus Multiple Collections
• Active Record Versus Mongoose
• ODM For Mongo and Rails