Welcome!

Enterprise DevOps, Log Management and Analytics

Sematext Blog

Subscribe to Sematext Blog: eMailAlertsEmail Alerts
Get Sematext Blog via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Latest Blogs from Sematext Blog
In this post we will: Introduce Kubernetes concepts and motivation for Kubernetes-aware monitoring and logging tooling Show how to deploy the Sematext Docker Agent to each Kubernetes node with DaemonSet Point out key Kubernetes metrics and log elements to help you troubleshoot and tune...
Docker is all the rage these days, but one doesn’t hear about running Solr on Docker very much. Last month, we gave a talk on the topic of running containerized Solr at the Lucene Revolution conference in Boston, the biggest open source conference dedicated to Apache Lucene/Solr....
Logging Libraries vs Log Shippers In the context of centralizing logs (say, to Logsene or your own Elasticsearch), we often get the question of whether one should log directly from the application (e.g. via an Elasticsearch or syslog appender) or use a dedicated log shipper. In this po...
This is a guest post by Nedim Šabić, developer of Fibratus, a tool for exploration and tracing of the Windows kernel.  Unlike Linux / UNIX environments which provide a plethora of open source and native tools to instrument the user / kernel space internals, the Windows operating system...
For this Black Friday, Sematext wishes you: more products sold more traffic and exposure more logs
This is a guest post by Prof. Chanwit Kaewkasi, Docker Captain who organized Swarm3K – the largest Docker Swarm cluster to date. Swarm3K Review Swarm3K was the second collaborative project trying to form a very large Docker cluster with the Swarm mode. It happened on 28th October...
Many of our clients use AWS EC2. In the context of Elasticsearch consulting or support, one question we often get is: should we use AWS Elasticsearch Service instead of deploying Elasticsearch ourselves? The question is valid whether “self hosted” means in EC2, some other cloud or your...
We are known worldwide for our Elasticsearch, ELK stack and Solr consulting services, and we are always happy to help others improve their skills in these technologies, not only through Solr & Elastic Stack trainings, but also by sharing our knowledge in meetups and conferences. Th...
Not everyone uses Splunk or ELK stack for logs. A few weeks ago, at the Lucene/Solr Revolution conference in Boston, we gave a talk about using Solr for logging, along with lots of good info about how to tune the logging pipeline. The talk also goes over the best AWS instance types, op...
SPM provides Akka monitoring via Kamon and has been doing that for quite a while now.  With SPM and Kamon you get out of the box metrics about Akka Actors, Dispatchers and Routers, about the JVMs your Akka app runs in, and system metrics. We’ve recently made a few nice improvements tha...
One of the things you learn when attending Sematext Solr training is how to scale Solr. We discuss various topics regarding leader shards and their replicas – things like when to go for more leaders, when to go for more replicas and when to go for both. We discuss what you can do...
SwarmZilla/swarm3k by Docker Captain Chanwit Kaewkasi is a unique community project/event aimed at launching a Docker Swarm cluster with 3000+ community-sponsored nodes. The previous project – Swarm2k – successfully demonstrated a 2000+ node Swarm cluster with only 3 Swarm ...
Sematext is hiring! More specifically, we are looking for people with Elasticsearch skills to join our Professional Services team. Our Elasticsearch Professional Services include: Consulting Production Support Training Most of our consulting work we do remotely, but occasionally a shor...
When it comes to centralizing logs to Elasticsearch, the first log shipper that comes to mind is Logstash. People hear about it even if it’s not clear what it does: – Bob: I’m looking to aggregate logs – Alice: you mean… like… Logstash? When you get into it, you realize cen...
RancherOS is one of the few “container only” operating systems and it evolved into an excellent orchestration tool for containers, competing e.g. with CoreOS. It supports several types of schedulers such as its own “Cattle” scheduler, as well as Kubernetes, Docker Swarm, and Mesos. A u...
In earlier posts, we explained how one can reindex data from one Elasticsearch cluster to another, or within the same Elasticsearch cluster, via tools like Logstash and rsyslog. The same recipes apply to Logsene, as it exposes the Elasticsearch API. Not only can you push data to Logsen...
As the world of software is growing, so is the ecosystem of DevOps tools and resources – for monitoring, for logging, for alerting, for continuous integration and deployment, configuration management, etc.  Nothing wrong with having lots of resources and tools, but here at Semate...
Docker is growing by leaps and bounds, and along with it its ecosystem. Being light, the predominant container deployment involves running just a single app or service inside each container. Most software products and services are made up of at least several such apps/services. We a...
Docker Datacenter (DDC) simplifies container orchestration and increases the flexibility and scalability of application deployments.  However, the high level of automation create new challenges for monitoring and log management. Organizations that introduce Docker Datacenter manage con...
Monitoring of Docker environments is challenging. Why? Because each container typically runs a single process, has its own environment, utilizes virtual networks, or has various methods of managing storage. Traditional monitoring solutions take metrics from each server and applications...
If you are running Elasticsearch in Docker, you may have flipped through our Running High Performance Fault-tolerant Elasticsearch Clusters on Docker slide deck.  Here is the video of the Running High Performance Fault-tolerant Elasticsearch Clusters on Docker talk given at Berlin Buzz...
If you are running Elasticsearch in Docker, here are some slides on that topic to make that smoother. See also Monitoring Official Elasticsearch Image on Docker  
SPM is one of the most comprehensive Kafka monitoring solutions, capturing some 200 Kafka metrics, including Kafka Broker, Producer, and Consumer metrics. While lots of those metrics are useful, there is one particular metric everyone wants to monitor – Consumer Lag. What is Cons...
Since we wrote about how to ship Heroku Logs to ELK we’ve received good feedback from Heroku users and, encouraged by that feedback, deployed a log ingestion service for apps running on Heroku. This makes it super easy to get structured Heroku Logs into Logsene, the hosted ELK logging ...
Next month, June 13-16, 2016, we will be running three Elastic Stack (aka ELK Stack) classes in New York City: June 13 & 14: Elasticsearch for Developers Training Workshop June 15: Elasticsearch Operations Training Workshop June 16: Elasticsearch for Logging Training Workshop All c...
If you’ve missed our Core Solr training in October 2015 in New York, here is another chance – we’re running the 2-day Core Solr class again next month – June 13 & 14, 2016. This course covers Solr 5.x as well as Solr 6.x!  You can see the complete course out...
Last time, when talking about Solr 6 we learned how to use streaming expressions to automatically update data in a collection. You can imagine this is not the only cool thing you can do with streaming expressions. Today, we will see how to re-index data in your collection for fields th...
Earlier on, we posted a recipe on reindexing data from within an Elasticsearch 2.3+ cluster. But this doesn’t work if you want to reindex in a different cluster or if your Elasticsearch is older than 2.3. Or both, when you’re trying to migrate from 1.x to 2.x or later. For such c...
One of the things that was extensively changed in Solr 6.0 are the Streaming Expressions and what we can do with them (hint: amazing stuff!). We already described Solr SQL support. Today, we’ll dig into the functionality that makes Solr SQL support possible – the Streaming ...
May 5 2016 — Sematext, a global, Brooklyn-based products and services company that builds innovative Cloud and On Premises solutions for application performance monitoring, log management and analytics, today announced that it has been recognized by Docker as the Ecosystem Technology P...
For those of you using Apache Kafka and Docker Cloud or considering it, we’ve got a Sematext user case study for your reading pleasure. In this use case, Ján Antala, a Software Engineer in the DevOps Team at @pygmalios, talks about the business and technical needs that drove their deci...
Last week, in the Solr 6, SolrCloud and SQL Queries post, we described how the recent release of Solr 6 in its SolrCloud mode is able to understand SQL. But this is not the only SolrCloud / Solr 6. Another addition that we can use is the Solr JDBC driver. We can use it just […]
Starting from Elasticsearch 5.0, you’ll be able to define pipelines within it that process your data, in the same way you’d normally do it with something like Logstash. We decided to take it for a spin and see how this new functionality (called Ingest) compares with Logstash filters in...
With the recent release of Solr 6.0, we got a host of new functionalities users have been anxiously waiting for. We’ve got the Parallel SQL over MapReduce that we recently blogged about, the new default similarity model, changes to the default similarity model configuration, the ...
With the recent release of Apache Lucene and Solr 6, we should familiarize ourselves with the juicy features that come with them. We have the new default Similarity implementation – BM25 – instead of the previously used TF-IDF Similarity, we have improvements in the default...
Docker Cloud is a hosted service for Docker Container Management, originally based on Tutum Cloud, which was acquired by Docker in October 2015. Sematext supported the deployment of Sematext Docker Agent on Tutum Cloud from the get-go, so naturally we were quick to add support for Dock...
In “Innovative Docker Log Management” we wrote about the alternative (and better?) method for Docker logging compared to log drivers, which do only log forwarding. Getting logs from Docker Containers collected, shipped and parsed out of the box is already a big time saver, ...
A while ago we published this post where we explained how you can get stats about rsyslog, such as the number of messages enqueued, the number of output errors and so on. The point was to send them to Elasticsearch (or Logsene, our logging SaaS, which exposes the Elasticsearch API) in ...
Just recently, we’ve described how to re-index your Elasticsearch data using the built-in re-index API in Elasticsearch 2.3 (and above). Today, we’ll look at another addition to the upcoming Elasticsearch v2.3+ – the Update by Query API. Yes, you got that right, you w...
Sooner or later, you’ll run into a problem of reindexing the data of your Elasticsearch instances. When we do Elasticsearch consulting for clients we always look at whether they have some way to efficiently reindex previously indexed data. The reasons for reindexing vary – ...