Welcome!

Enterprise DevOps, Log Management and Analytics

Sematext Blog

Subscribe to Sematext Blog: eMailAlertsEmail Alerts
Get Sematext Blog via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Latest Blogs from Sematext Blog
On March 29-30, Cloud NativeCon + KubeCon – Berlin, Germany brings together almost 1500 developers, architects, technical leaders, CIOs, CTOs, and analysts from open source cloud native communities – all under a single roof. Come and learn about Fluentd, Kubernetes, […]
Sematext Solr AutoComplete is an open-source Solr add-on that provides suggest-as-you-type functionality. In this post we’ll explain how you can install it, load the autocomplete collection/core with suggestions and how to run queries to get those suggestions back. Why Sematext [...
In the world of DevOps, metric collection, log centralization and analysis Apache Kafka is the most commonly used middleware.  More specifically, it is used as a fast, persistent queue between data sources like log shippers and the storage that makes […]
Since we work with Elasticsearch and Solr on a daily basis and know so much about them we like helping others improve their knowledge of these technologies, not only through Solr & Elastic Stack trainings, but also by sharing our […]
Intrusion detection systems generate highly valuable logs with network usage details and alerts. They collect vast amounts of data and typically store them in structures with a large number of fields.  To make sense of so much data and to […]
As you know, in Sematext we looooove logs and metrics and we enjoy playing with them on a daily basis. We have our Logsene, which is all about logs and their analysis. We have our SPM which can monitor half […]
When dealing with log centralization in your organization you have to start with something. Often times people start by collecting logs for the most crucial pieces of software, and frequently one chooses to ship them to their own in-house Elasticsearch-based solution (aka ELK stack) or...
Pretty much everyone’s heard about syslog: with its roots in the 80s, it’s still used for a lot of the logging done today. Mostly because of its long history, syslog is quite a vague concept, referring to many things. Which is why you’ve probably heard: Check syslog, ...
Enterprises without DevOps teams and culture typically experience long applications deployment cycles. Once developers make a new release the operations team needs to deploy it to one of the existing or new servers, and it often takes a long time (up to several weeks is not uncommon) t...
The recent ransom attack on public Elasticsearch instances showed that Elasticsearch security is still a hot topic. Elasticsearch was not the only target – tens of thousands of poorly configured MongoDB databases have been compromised over the past week, too, compromising over 27...
One way to create a better search experience is to understand the user intent.  One of the phases in that process is query understanding, and one simple step in that direction is query segmentation. In this post, we’ll cover what query segmentation is and when it is useful. We will als...
2017 is almost here and, like last year, we thought we’d share how 2016 went for us.  We remain committed to be your “one-stop shop” for all things Elasticsearch and Solr: from Consulting, Production Support, and Training, to complementing that with our Logsene for all your logs, and S...
Nowadays there are more and more organizations searching for fault-tolerant and highly available solutions for various parts of their infrastructure, including search, which evolved from merely a “nice to have” feature to the first class citizen and a “must have”...
Running on Elasticsearch on Docker sounds like a natural fit – both technologies promise elasticity. However, running a truly elastic Elasticsearch cluster on Docker Swarm became somewhat difficult with Docker Swarm 1.2. Why? Since Elasticsearch gave up on multicast discovery (by...
Back in 2011 – more than half a decade ago(!) – we’ve reviewed Top JavaScript Dynamic Table Libraries.  Clearly, a lot has changed since then.  Earlier this year, we started reworking our SPM & Logsene front-ends, building them on top of ReactJS, Redux, and ES6.  In the...
In this post we will: Introduce Kubernetes concepts and motivation for Kubernetes-aware monitoring and logging tooling Show how to deploy the Sematext Docker Agent to each Kubernetes node with DaemonSet Point out key Kubernetes metrics and log elements to help you troubleshoot and tune...
Docker is all the rage these days, but one doesn’t hear about running Solr on Docker very much. Last month, we gave a talk on the topic of running containerized Solr at the Lucene Revolution conference in Boston, the biggest open source conference dedicated to Apache Lucene/Solr....
Logging Libraries vs Log Shippers In the context of centralizing logs (say, to Logsene or your own Elasticsearch), we often get the question of whether one should log directly from the application (e.g. via an Elasticsearch or syslog appender) or use a dedicated log shipper. In this po...
This is a guest post by Nedim Šabić, developer of Fibratus, a tool for exploration and tracing of the Windows kernel.  Unlike Linux / UNIX environments which provide a plethora of open source and native tools to instrument the user / kernel space internals, the Windows operating system...
For this Black Friday, Sematext wishes you: more products sold more traffic and exposure more logs
This is a guest post by Prof. Chanwit Kaewkasi, Docker Captain who organized Swarm3K – the largest Docker Swarm cluster to date. Swarm3K Review Swarm3K was the second collaborative project trying to form a very large Docker cluster with the Swarm mode. It happened on 28th October...
Many of our clients use AWS EC2. In the context of Elasticsearch consulting or support, one question we often get is: should we use AWS Elasticsearch Service instead of deploying Elasticsearch ourselves? The question is valid whether “self hosted” means in EC2, some other cloud or your...
We are known worldwide for our Elasticsearch, ELK stack and Solr consulting services, and we are always happy to help others improve their skills in these technologies, not only through Solr & Elastic Stack trainings, but also by sharing our knowledge in meetups and conferences. Th...
Not everyone uses Splunk or ELK stack for logs. A few weeks ago, at the Lucene/Solr Revolution conference in Boston, we gave a talk about using Solr for logging, along with lots of good info about how to tune the logging pipeline. The talk also goes over the best AWS instance types, op...
SPM provides Akka monitoring via Kamon and has been doing that for quite a while now.  With SPM and Kamon you get out of the box metrics about Akka Actors, Dispatchers and Routers, about the JVMs your Akka app runs in, and system metrics. We’ve recently made a few nice improvements tha...
One of the things you learn when attending Sematext Solr training is how to scale Solr. We discuss various topics regarding leader shards and their replicas – things like when to go for more leaders, when to go for more replicas and when to go for both. We discuss what you can do...
SwarmZilla/swarm3k by Docker Captain Chanwit Kaewkasi is a unique community project/event aimed at launching a Docker Swarm cluster with 3000+ community-sponsored nodes. The previous project – Swarm2k – successfully demonstrated a 2000+ node Swarm cluster with only 3 Swarm ...
Sematext is hiring! More specifically, we are looking for people with Elasticsearch skills to join our Professional Services team. Our Elasticsearch Professional Services include: Consulting Production Support Training Most of our consulting work we do remotely, but occasionally a shor...
When it comes to centralizing logs to Elasticsearch, the first log shipper that comes to mind is Logstash. People hear about it even if it’s not clear what it does: – Bob: I’m looking to aggregate logs – Alice: you mean… like… Logstash? When you get into it, you realize cen...
RancherOS is one of the few “container only” operating systems and it evolved into an excellent orchestration tool for containers, competing e.g. with CoreOS. It supports several types of schedulers such as its own “Cattle” scheduler, as well as Kubernetes, Docker Swarm, and Mesos. A u...
In earlier posts, we explained how one can reindex data from one Elasticsearch cluster to another, or within the same Elasticsearch cluster, via tools like Logstash and rsyslog. The same recipes apply to Logsene, as it exposes the Elasticsearch API. Not only can you push data to Logsen...
As the world of software is growing, so is the ecosystem of DevOps tools and resources – for monitoring, for logging, for alerting, for continuous integration and deployment, configuration management, etc.  Nothing wrong with having lots of resources and tools, but here at Semate...
Docker is growing by leaps and bounds, and along with it its ecosystem. Being light, the predominant container deployment involves running just a single app or service inside each container. Most software products and services are made up of at least several such apps/services. We a...
Docker Datacenter (DDC) simplifies container orchestration and increases the flexibility and scalability of application deployments.  However, the high level of automation create new challenges for monitoring and log management. Organizations that introduce Docker Datacenter manage con...
Monitoring of Docker environments is challenging. Why? Because each container typically runs a single process, has its own environment, utilizes virtual networks, or has various methods of managing storage. Traditional monitoring solutions take metrics from each server and applications...
If you are running Elasticsearch in Docker, you may have flipped through our Running High Performance Fault-tolerant Elasticsearch Clusters on Docker slide deck.  Here is the video of the Running High Performance Fault-tolerant Elasticsearch Clusters on Docker talk given at Berlin Buzz...
If you are running Elasticsearch in Docker, here are some slides on that topic to make that smoother. See also Monitoring Official Elasticsearch Image on Docker  
SPM is one of the most comprehensive Kafka monitoring solutions, capturing some 200 Kafka metrics, including Kafka Broker, Producer, and Consumer metrics. While lots of those metrics are useful, there is one particular metric everyone wants to monitor – Consumer Lag. What is Cons...
Since we wrote about how to ship Heroku Logs to ELK we’ve received good feedback from Heroku users and, encouraged by that feedback, deployed a log ingestion service for apps running on Heroku. This makes it super easy to get structured Heroku Logs into Logsene, the hosted ELK logging ...
Next month, June 13-16, 2016, we will be running three Elastic Stack (aka ELK Stack) classes in New York City: June 13 & 14: Elasticsearch for Developers Training Workshop June 15: Elasticsearch Operations Training Workshop June 16: Elasticsearch for Logging Training Workshop All c...